Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object’s position or contour, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm relies on optical systems that—instead of performing imaging—act as encoders that optically compress images into low-dimensional spaces by extracting salient features; however, the performance of these encoders is typically limited by their linearity. Here we report a nonlinear, multilayer optical neural network (ONN) encoder for image sensing based on a commercial image intensifier as an optical-to-optical nonlinear activation function. This nonlinear ONN outperforms similarly sized linear optical encoders across several representative tasks, including machine-vision benchmarks, flow-cytometry image classification and identification of objects in a three-dimensionally printed real scene. For machine-vision tasks, especially those featuring incoherent broadband illumination, our concept allows for a considerable reduction in the requirement of camera resolution and electronic post-processing complexity. In general, image pre-processing with ONNs should enable image-sensing applications that operate accurately with fewer pixels, fewer photons, higher throughput and lower latency.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017).

Ballard, Z., Brown, C., Madni, A. M. & Ozcan, A. Machine learning and computation-enabled intelligent sensor design. Nat. Mach. Intell. 3, 556–565 (2021).

AdvanceLEDSupply

Bandyopadhyay, S. et al. Single chip photonic deep neural network with accelerated training. Preprint at https://arxiv.org/abs/2208.01623 (2022).

Colburn, S., Chu, Y., Shilzerman, E. & Majumdar, A. Optical frontend for a convolutional neural network. Appl. Optics 58, 3179–3186 (2019).

Lee, K. C. M., Guck, J., Goda, K. & Tsia, K. K. Toward deep biophysical cytometry: prospects and challenges. Trends Biotechnol. 39, 1249–1262 (2021).

Heuser, T. et al. Developing a photonic hardware platform for brain-inspired computing based on 5 × 5 VCSEL arrays. J. Phys. Photon. 2, 44002 (2020).

Guo, Q. et al. Femtojoule femtosecond all-optical switching in lithium niobate nanophotonics. Nat. Photon. 16, 625–631 (2022).

Li, G. H.et al. All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning. Nanophotonics https://doi.org/10.1515/nanoph-2022-0137 (2022).

Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 607, 501–506 (2022).

Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Optuna: a next-generation hyperparameter optimization framework. In Proc. 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2623–2631 (ACM, 2019).

Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006).

We wish to thank NTT Research for their financial and technical support (to T.W., L.G.W., S.-Y.M., T.O. and P.L.M.). Portions of this work were supported by the National Science Foundation (award no. CCF-1918549 to T.W., M.M. Stein and P.L.M.), a Kavli Institute at Cornell instrumentation grant (to T.W. and P.L.M.), and a David and Lucile Packard Foundation Fellowship (to P.L.M.). P.L.M. acknowledges membership of the CIFAR Quantum Information Science Program as an Azrieli Global Scholar. T.W. acknowledges the partial support from Schmidt Futures via an Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship to Cornell University. We acknowledge helpful discussions with A. Senanian, B. Malia, F. Presutti, V. Kremenetski, S. Prabhu, A. Barth, R. Oliver, and D. Schraivogel. We also acknowledge S. Sohoni for help with figure design.

Zhou, T. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photon. 15, 367–373 (2021).

Fard, M. M. P. et al. Experimental realization of arbitrary activation functions for optical neural networks. Opt. Express 28, 12138–12148 (2020).

Lin, H. W., Tegmark, M. & Rolnick, D. Why does deep and cheap learning work so well? J. Stat. Phys. 168, 1223–1247 (2017).

Advanced LEDGrowLights

Kubala, K., Dowski, E. & Cathey, W. T. Reducing complexity in computational imaging systems. Opt. Express 11, 2102–2108 (2003).

AdvancedLighting

Baek, S.-H. et al. Single-shot hyperspectral-depth imaging with learned diffractive optics. In Proc. IEEE/CVF International Conference on Computer Vision 2651–2660 (IEEE, 2021).

T.W., L.G.W., M.M. Sohoni and P.L.M. conceived the project and designed the experiments. M.M. Sohoni and T.W. built and performed the experiments on the nonlinear and linear ONN encoders, and analysed the data. T.W. performed the extended cell-organelle simulations. M.M. Stein performed the neural architecture search for QuickDraw reconstruction. S-Y.M. and T.O. aided in simulations of deep optical encoders. M.G.A. assisted with 3D-scene modelling. L.G.W., T.W., M.M. Sohoni and P.L.M. wrote the manuscript. P.L.M. and L.G.W. supervised the project.

Asif, M. S., Ayremlou, A., Sankaranarayanan, A., Veeraraghavan, A. & Baraniuk, R. G. Flatcam: thin, lensless cameras using coded aperture and computation. IEEE Trans. Comput. Imaging 3, 384–397 (2016).

Stork, D. G. & Robinson, M. D. Theoretical foundations for joint digital-optical analysis of electro-optical imaging systems. Appl. Opt. 47, B64–B75 (2008).

Markley, E., Liu, F. L., Kellman, M., Antipa, N. & Waller, L. Physics-based learned diffuser for single-shot 3D imaging. In NeurIPS 2021 Workshop on Deep Learning and Inverse Problems (NeurIPS, 2021).

LED Advanced

Martel, J. N. P., Mueller, L. K., Carey, S. J., Dudek, P. & Wetzstein, G. Neural sensors: learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Trans. Pattern Anal. Mach. Intell. 42, 1642–1653 (2020).

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (IEEE, 2016).

Vargas, E., Martel, J. N. P., Wetzstein, G. & Arguello, H. Time-multiplexed coded aperture imaging: learned coded aperture and pixel exposures for compressive imaging systems. In Proc. IEEE/CVF International Conference on Computer Vision 2692–2702 (IEEE, 2021).

The demonstration data for data gathering, as well as training data for the all-optical/digital neural networks, are available at https://github.com/mcmahon-lab/Image-sensing-with-multilayer-nonlinear-optical-neural-networks.

Ryou, A. et al. Free-space optical neural network based on thermal atomic nonlinearity. Photon. Res. 9, B128–B134 (2021).

Sitzmann, V. et al. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 1–13 (2018).

Matic, R. M. & Goodman, J. W. Comparison of optical predetection processing and postdetection linear processing for partially coherent image estimation. J. Opt. Soc. Am. A 6, 213–228 (1989).

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR, 2019).

Advancedlighting technology uk limited

Nahmias, M. A., Shastri, B. J., Tait, A. N. & Prucnal, P. R. A leaky integrate-and-fire laser neuron for ultrafast cognitive computing. IEEE J. Sel. Top. Quantum Electron. 19, 1–12 (2013).

Gibson, G. M., Johnson, S. D. & Padgett, M. J. Single-pixel imaging 12 years on: a review. Opt. Express 28, 28190–28208 (2020).

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Makarenko, M. et al. Real-time hyperspectral imaging in hardware via trained metasurface encoders. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12692–12702 (IEEE, 2022); https://doi.org/10.1109/CVPR52688.2022.01236

Mirek, R. et al. Neural networks based on ultrafast time-delayed effects in exciton polaritons. Phys. Rev. Appl. 17, 054037 (2022).

Nature Photonics thanks Jacques Carolan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Narayan, A., Berger, B. & Cho, H. Assessing single-cell transcriptomic variability through density-preserving data visualization. Nat. Biotechnol. 39, 765–774 (2021).

Jongejan, J., Rowley, H., Kawashima, T., Kim, J. & Fox-Gieg, N. The Quick, Draw! AI Experiment https://quickdraw.withgoogle.com/ (2016).

Wang, T., Sohoni, M.M., Wright, L.G. et al. Image sensing with multilayer nonlinear optical neural networks. Nat. Photon. 17, 408–415 (2023). https://doi.org/10.1038/s41566-023-01170-8

Li, Y. et al. Deep cytometry: deep learning with real-time inference in cell sorting and flow cytometry. Sci. Rep. 9, 1–12 (2019).

Pad, P. et al. Efficient neural vision systems based on convolutional image acquisition. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12285–12294 (IEEE, 2020).

Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8, 1–10 (2018).

Kim, K., Konda, P. C., Cooke, C. L., Appel, R. & Horstmeyer, R. Multi-element microscope optimization by a learned sensing network with composite physical layers. Opt. Lett. 45, 5684–5687 (2020).

T.W., M.M. Sohoni, L.G.W. and P.L.M. are listed as inventors on a US provisional patent application (serial no. 63/392,042) on nonlinear optical neural network pre-processors for imaging and image sensing. The other authors declare no competing interests.

Tianyu Wang, Mandar M. Sohoni, Logan G. Wright, Martin M. Stein, Shi-Yuan Ma, Tatsuhiro Onodera, Maxwell G. Anderson & Peter L. McMahon

Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).