Differentiable imaging

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) This article may rely excessively on sources too closely associated with the subject, potentially preventing the article from being verifiable and neutral. Please help improve it by replacing them with more appropriate citations to reliable, independent sources. (September 2025) (Learn how and when to remove this message) This article may incorporate text from a large language model. It may include hallucinated information or fictitious references. Copyright violations or claims lacking verification should be removed. Please see the associated project page for additional guidance. (September 2025) (Learn how and when to remove this message) The examples and perspective in this are too limited. Of the 9 refs (9/23/2025), 7 are from one group. Digital modelling of imaging in electron microscopes goes back to the 1980's, see multislice for instance; I know it is there for x-ray and optical, and the masses of tomography methods. This page should acknowledge at least some of the broader context. Thus may not include all significant viewpoints. Please improve the article or discuss the issue. (September 2024) (Learn how and when to remove this message) (Learn how and when to remove this message)

Differentiable imaging is a method within computational imaging that incorporates differentiable programming to design imaging systems. It treats the entire imaging process - from light passing through optical components to the numerical reconstruction—as a differentiable programming problem. This approach links optical hardware with numerical reconstruction, enabling joint optimization of both parts through differentiable programming.^[1] Differentiable imaging additionally extends the scope of computational imaging beyond image reconstruction, such as by aiding in characterization of optical components.^[2]

Computational imaging combines optical hardware and computational algorithms to capture and reconstruct information that conventional imaging system cannot. This is achieved from a combination of the imaging system and the software used in the image reconstruction.^[3] Since the captured information may not directly show the image of the target, these systems often rely on numerical models that describe how light encodes the target. In practice, such models may deviate from the physical systems due to uncertainties such as noise, misalignments, manufacturing imperfections, environmental variations, and etc. These uncertainties can cause a mismatch between the physical system and its numerical model, which may degrade reconstruction quality and limit the effectiveness of the hardware–software co-design.^[3] Uncertainty quantification is also studied in other hybrid physical–numerical systems, such as digital twin.^[4] While numerical modeling imaging systems date back to the several decades, such as the multislice method in electron microscopy^[5] or X-Ray nanotomography^[6], differentiable imaging emphasizes jointly modeling uncertainties and solving inverse problems with image reconstruction simultaneously.

Differentiable imaging transforms the traditional encoding model ${\textstyle y=f(x)}$ into a more comprehensive formulation ${\textstyle y=f(x,\theta )}$, where ${\displaystyle \theta }$ represents a parameter set of mismatches between physical systems and numerical models.

The forward model captures the entire imaging pipeline through a series of interconnected component functions:

${\displaystyle y=f(x,\theta ),\qquad f=f_{noise}\circ f_{c}\circ f_{oc}\circ f_{x}\circ f_{oi}\circ f_{i},}$

where the function composition operator ${\displaystyle \circ }$ connects each system component, and ${\displaystyle \theta =\{\theta _{c},\theta _{oc},\ldots \}}$ encompasses uncertainty system parameters. Each component corresponds to specific physical processes within the imaging system, from illumination through object interactions to sensor behavior and noises.

This forward model enables the formulation of an inverse problem that simultaneously optimizes system parameters while reconstructing images:

${\displaystyle x^{*},\theta ^{*}={\text{argmin}}_{x,\theta }{\mathcal {L}}(f(x,\theta ),y)+\sum _{n=1}^{N}\beta _{n}{\mathcal {R}}_{n}(x)}$

${\displaystyle s.t.\quad x\in \Omega _{x},\theta \in \Omega _{\theta }}$

Here, ${\displaystyle {\mathcal {L}}(f(x,\theta ),y)}$ represents the fidelity term that quantifies the discrepancy between the model predictions and measured data.

The whole process of the ${\displaystyle y=f(x,\theta )}$ is constructed as a computer graph based on differentiable programming, and the inverse problem is solved with gradient based algorithm, while the gradient is calculated with automatic differentiation.

One application of differentiable imaging is uncertainty management, which seeks to quantify and mitigate the impact of factors induce reality-numerical mismatch. Explicitly accounting for uncertainties can improve reconstruction accuracy and system robustness. Examples include:

• Model-related uncertainties: unknown or unmeasurable variables—for instance, optical system quantities that differ from the design specifications^{[7][8][9][10][11]}
• Data and system uncertainties: artifacts introduced during image acquisition, such as low-quality data, noise, or hardware imperfections^[12][13]
• Manufacturing uncertainties: variability in the production of imaging hardware—such as slight deviations in lens curvature or sensor alignment—that alters the physical system's behavior^[14]

As of this edit, this article uses content from "Differentiable Imaging: Progress, Challenges, and Outlook", authored by Ni Chen, David Brady, and Edmund Lam, which is licensed in a way that permits reuse under the Creative Commons Attribution-ShareAlike 4.0 International License, but not under the GFDL. All relevant terms must be followed.