CuNeRF is the first zero-shot Medical Image Arbitrary-Scale Super Resolution framework. After training on an LR medical volume (a) itself, CuNeRF can build the corresponding continuous volumetric representation (b), which is able to achieve (c) Free-viewpoint slice synthesis: yielding novel-view medical slices from the arbitrary viewpoints, and (d) Arbitrary-scale super-resolution: upsampling medical images at arbitrary scales in a continuous domain.

Medical image arbitrary-scale super-resolution (MIASSR) has recently gained widespread attention, aiming to supersample medical volumes at arbitrary scales via a single model. However, existing MIASSR methods face two major limitations: (i) reliance on high-resolution (HR) volumes and (ii) limited generalization ability, which restricts their applications in various scenarios. To overcome these limitations, we propose Cube-based Neural Radiance Field (CuNeRF), a zero-shot MIASSR framework that is able to yield medical images at arbitrary scales and free viewpoints in a continuous domain. Unlike existing MISR methods that only fit the mapping between low-resolution (LR) and HR volumes, CuNeRF focuses on building a continuous volumetric representation from each LR volume without the knowledge from the corresponding HR one. This is achieved by the proposed differentiable modules: cube-based sampling, isotropic volume rendering, and cube-based hierarchical rendering. Through extensive experiments on magnetic resource imaging (MRI) and computed tomography (CT) modalities, we demonstrate that CuNeRF can synthesize high-quality SR medical images, which outperforms state-of-the-art MISR methods, achieving better visual verisimilitude and fewer objectionable artifacts. Compared to existing MISR methods, our CuNeRF is more applicable in practice.

Motivation

NeRF (a) only samples the rays (yellow arrow) corresponding to each training pixel (red circle), which cannot cover the whole representation fields, leaving some "holes" (i.e., unmodeled spaces) between adjacent training pixels. To address this issue, CuNeRF (b) samples cubes (purple cube) centered by each training pixel, and therefore the holes are well-covered by the spatial overlaps. (c) Visual comparisons of 3D MISR at $\times$2.5 upsampling factor show that NeRF yields grid-like artifacts, while ArSSR produces non-existent details. By contrast, our CuNeRF achieves better visual verisimilitude and fewer artifacts.

Framework

The overall framework of our CuNeRF. To synthesize a pixel (red circle) with the spatial position $\mathbf{x}_t=(x, y, z)$, (a) CuNeRF first uniformly samples $N$ points as a point set $\{\hat{\mathbf{x}}_i\}^N_{i=1}$ within the cube space (purple cube) centered by $\mathbf{x}_t$. Then, CuNeRF obtains the coarse estimation (blue cube) by feeding the sampling points into an MLP $F_{\Theta}$ to produce the set of corresponding pixel intensity $\{c_i\}_{i=1}^N$ and volume density $\{\sigma_i\}_{i=1}^N$. (b) Subsequently, assuming $\sigma$ of each sampling point is only related to the distance with the cube center $\mathbf{x}_t$, CuNeRF computes the coarse output of the target pixel via volume integral. (c) Finally, CuNeRF resamples the points under the probability density function (PDF) of coarse estimation to acquire the fine estimation (orange cube) of the cube. The fine output is generated by the same procedures as (b). Since these two rendering functions are differentiable, CuNeRF can be optimized by minimizing the rendering loss $\mathcal{L}_{A}$. The fine output is the final rendering result of the target spatial position $\mathbf{x}_t$.