[{"content":"Gemini Team, Google DeepMind (incl. J. Adler) · arXiv preprint (2025)\nThe Gemini 2.5 family, pushing the frontier on reasoning, multimodality, long context and next-generation agentic capabilities.\nPreprint\n","date":"17 June 2025","permalink":"https://jonasadler.com/publications/gemini-2-5/","section":"Publications","summary":"The Gemini 2.5 model family, advancing reasoning, multimodality, long context and agentic capabilities.","title":"Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities"},{"content":"Biography #I work at the intersection of machine learning and the sciences, contributing to AlphaFold and to Google\u0026rsquo;s Gemini models. Before DeepMind I completed a PhD in applied mathematics at KTH, on inverse problems and deep learning, and was a research scientist at Elekta.\nThe best way to reach me is by email at contact@jonasadler.com.\nNews # Dec 2024: The work behind AlphaFold was awarded the 2024 Nobel Prize in Chemistry (to John Jumper and Demis Hassabis). I was part of the AlphaFold team and joined the celebrations in Stockholm. May 2024: AlphaFold 3 published in Nature — extending accurate structure prediction to biomolecular interactions. Dec 2023: First Gemini models released. Jul 2021: AlphaFold published in Nature, alongside structures for the human proteome. ","date":null,"permalink":"https://jonasadler.com/","section":"Home","summary":"\u003ch2 id=\"biography\" class=\"relative group\"\u003eBiography \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#biography\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cp\u003eI work at the intersection of machine learning and the sciences, contributing to \u003ca href=\"https://jonasadler.com/publications/alphafold/\"\u003eAlphaFold\u003c/a\u003e and to Google\u0026rsquo;s \u003ca href=\"https://jonasadler.com/publications/gemini/\"\u003eGemini\u003c/a\u003e models. Before DeepMind I completed a PhD in applied mathematics at KTH, on inverse problems and deep learning, and was a research scientist at Elekta.\u003c/p\u003e\n\u003cp\u003eThe best way to reach me is by email at \u003ca href=\"mailto:contact@jonasadler.com\"\u003econtact@jonasadler.com\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 id=\"news\" class=\"relative group\"\u003eNews \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#news\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDec 2024:\u003c/strong\u003e The work behind \u003ca href=\"https://jonasadler.com/publications/alphafold/\"\u003eAlphaFold\u003c/a\u003e was awarded the \u003cstrong\u003e2024 Nobel Prize in Chemistry\u003c/strong\u003e (to John Jumper and Demis Hassabis). I was part of the AlphaFold team and joined the celebrations in Stockholm.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMay 2024:\u003c/strong\u003e \u003ca href=\"https://jonasadler.com/publications/alphafold3/\"\u003eAlphaFold 3\u003c/a\u003e published in \u003cem\u003eNature\u003c/em\u003e — extending accurate structure prediction to biomolecular interactions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDec 2023:\u003c/strong\u003e First \u003ca href=\"https://jonasadler.com/publications/gemini/\"\u003eGemini\u003c/a\u003e models released.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eJul 2021:\u003c/strong\u003e \u003ca href=\"https://jonasadler.com/publications/alphafold/\"\u003eAlphaFold\u003c/a\u003e published in \u003cem\u003eNature\u003c/em\u003e, alongside \u003ca href=\"https://jonasadler.com/publications/alphafold-human-proteome/\"\u003estructures for the human proteome\u003c/a\u003e.\u003c/li\u003e\n\u003c/ul\u003e","title":"Home"},{"content":"","date":null,"permalink":"https://jonasadler.com/publications/","section":"Publications","summary":"","title":"Publications"},{"content":"J. Abramson, J. Adler, J. Dunger, R. Evans, T. Green, A. Pritzel, … J. M. Jumper · Nature (2024)\nA substantially updated AlphaFold model that predicts the joint structure of complexes spanning proteins, nucleic acids, small molecules, ions and modified residues with greatly improved accuracy over specialised tools.\nArticle · Code\n","date":"8 May 2024","permalink":"https://jonasadler.com/publications/alphafold3/","section":"Publications","summary":"AlphaFold 3 — accurate prediction of the structure of complexes including proteins, nucleic acids, ligands and more. (Nature)","title":"Accurate structure prediction of biomolecular interactions with AlphaFold 3"},{"content":"Gemini Team, Google DeepMind (incl. J. Adler) · arXiv preprint (2024)\nGemini 1.5 extends multimodal understanding to context windows of millions of tokens, with near-perfect retrieval across long documents, video and audio.\nPreprint\n","date":"8 March 2024","permalink":"https://jonasadler.com/publications/gemini-1-5/","section":"Publications","summary":"A compute-efficient multimodal model with recall and reasoning over millions of tokens of context.","title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context"},{"content":"Gemini Team, Google DeepMind (incl. J. Adler) · arXiv preprint (2023)\nThe first Gemini models — a family of natively multimodal models with strong capabilities across image, audio, video and text understanding.\nPreprint\n","date":"6 December 2023","permalink":"https://jonasadler.com/publications/gemini/","section":"Publications","summary":"Google DeepMind\u0026rsquo;s family of natively multimodal models across text, images, audio, video and code.","title":"Gemini: A Family of Highly Capable Multimodal Models"},{"content":"S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y. Ganin, P. H. Richemond, … J. Adler, … et al. · arXiv preprint (2022)\nContinuous Diffusion for Categorical Data (CDCD) adapts continuous diffusion models to discrete data such as language by diffusing in embedding space with a score-interpolation objective.\nPreprint\n","date":"28 November 2022","permalink":"https://jonasadler.com/publications/continuous-diffusion/","section":"Publications","summary":"A framework (CDCD) for modelling categorical data such as text with continuous diffusion models.","title":"Continuous diffusion for categorical data"},{"content":"K. Tunyasuvunakool, J. Adler, Z. Wu, T. Green, M. Zielinski, A. Žídek, … D. Hassabis · Nature (2021)\nAlphaFold applied at proteome scale, providing accurate structure predictions covering the human proteome and 20 other organisms, released openly through the AlphaFold Protein Structure Database.\nArticle\n","date":"22 July 2021","permalink":"https://jonasadler.com/publications/alphafold-human-proteome/","section":"Publications","summary":"Applying AlphaFold at scale to predict structures across the entire human proteome. (Nature)","title":"Highly accurate protein structure prediction for the human proteome"},{"content":"J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, … J. Adler, … D. Hassabis · Nature (2021)\nAlphaFold predicts protein structures with atomic accuracy directly from sequence, recognised as a solution to the 50-year-old protein folding problem at CASP14.\nArticle · Code\n","date":"15 July 2021","permalink":"https://jonasadler.com/publications/alphafold/","section":"Publications","summary":"AlphaFold — the deep learning system that solved the protein structure prediction problem. (Nature)","title":"Highly accurate protein structure prediction with AlphaFold"},{"content":"Sebastian Banert, Jevgenija Rudzusika, Ozan Öktem, Jonas Adler · arXiv\nWe propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set using some predefined method, we train a deep neural network to pick the best update. Finally, we show that the method is applicable to several cases of smooth and non-smooth optimization and show superior results to established accelerated solvers.\nPDF · Preprint · Code\n","date":"12 May 2021","permalink":"https://jonasadler.com/publications/accelerated_fb/","section":"Publications","summary":"Provably convergent convex optimization using Deep Learning.","title":"Accelerated Forward-Backward Optimization using Deep Learning"},{"content":"GAMM · Vienna, Austria\nDeep learning has shown tremendous success in solving various tasks in several fields of science such as in image and natural language processing. Recently, it has also been applied to solve inverse problems and empirical evidence in image reconstruction points to drastic improvements over classical approaches. A natural question that arrises is: how good can it actually get? We try to answer this by characterizing the optimal solution given various loss functions. In addition to this, we show that these upper bounds are empirically approximated quite well using deep neural networks.\nPDF\n","date":"18 February 2019","permalink":"https://jonasadler.com/talks/2019gamm/","section":"Talks","summary":"GAMM — Vienna, Austria","title":"Invited Talk: Deep learning for inverse problems. Where are we, and how far can we go?"},{"content":"","date":null,"permalink":"https://jonasadler.com/talks/","section":"Talks","summary":"","title":"Talks"},{"content":"BASP Frontiers · Villars-sur-Ollon, Switzerland\nWe show how generative models from machine learning can be used to perform Bayesian inversion on large-scale problems. Specifically, we train a Wasserstein GAN on 3D CT data and use it to sample from the posterior distribution. We demonstrate the ability of such Bayesian inversion by computing estimators and performing a hypothesis test.\nPDF\n","date":"4 February 2019","permalink":"https://jonasadler.com/talks/2019basp/","section":"Talks","summary":"BASP Frontiers — Villars-sur-Ollon, Switzerland","title":"Invited Talk: Deep Posterior Sampling"},{"content":"Deep Learning and Inverse Problems · Stockholm, Sweden\nThe ability to better characterize statistical properties of solutions to many inverse problems is essential for decision making. Bayesian inversion offers a tractable framework for such an analysis, but current approaches are computationally unfeasible for most realistic imaging applications in the clinic and asymptotic characterizations rely on unrealistic assumptions.\\n We show how deep learning can be used for Bayesian inversion by introducing two novel methods: a sampling based method using GANs and a direct approach that trains a neural network using a novel loss function. We demonstrate the capabilities of both methods by performing uncertainty quantification on ultra low dose 3D helical CT. We estimate the posterior mean and standard deviation of the 3D images and furthermore perform a Bayesian hypothesis test to assess the presence of a \u0026lsquo;dark spot\u0026rsquo; in the liver of a cancer stricken patient.\nPDF\n","date":"22 January 2019","permalink":"https://jonasadler.com/talks/2019dlip/","section":"Talks","summary":"Deep Learning and Inverse Problems — Stockholm, Sweden","title":"Contributed Talk: Deep Bayesian Inversion"},{"content":"Seminar at New York University · New York, USA\nThe talk will outline recent approaches for using (deep) convolutional neural networks to solve a wide range of inverse problems, such as image reconstruction in medical tomography. A key element is to use a neural network architecture for reconstruction that includes physics based models that describe how data is generated as well as its statistical properties Another is the possibility to integrate complex task related a priori information and elements of decision making into the reconstruction procedure. The resulting approach outperforms current state-of-the-art in terms of \u0026lsquo;quality\u0026rsquo;, computational speed and there is no need to manually set parameters as with variational methods. Furthermore, the amount of training data and network size can be kept surprisingly small. The talk will also touch upon further developments based on using generative adversarial networks for uncertainty quantification.\nPDF\n","date":"12 December 2018","permalink":"https://jonasadler.com/talks/2018nyu/","section":"Talks","summary":"Seminar at New York University — New York, USA","title":"Invited Talk: Recent advances in using machine learning for image reconstruction"},{"content":"Medical imaging meets NeurIPS · Montreal, Canada\nWe consider the problem of performing a task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. Several such tasks have been approached using deep neural networks, and recent advancements in image reconstruction using learned iterative schemes now enable us to have a fully differentiable, end-to-end trainable, imaging pipeline. The suggested framework is adaptable, with a plug-and-play structure for adjusting to both the inverse problem and the task at hand. The approach is demonstrated on joint tomographic image reconstruction and semantic segmentation.\nPDF\n","date":"8 December 2018","permalink":"https://jonasadler.com/talks/2018medneurips/","section":"Talks","summary":"Medical imaging meets NeurIPS — Montreal, Canada","title":"Oral: Task adapted reconstruction for inverse problems"},{"content":"Jonas Adler, Ozan Öktem · arXiv\nCharacterizing statistical properties of solutions of inverse problems is essential for decision making. Bayesian inversion offers a tractable framework for this purpose, but current approaches are computationally unfeasible for most realistic imaging applications in the clinic. We introduce two novel deep learning based methods for solving large-scale inverse problems using Bayesian inversion: a sampling based method using a WGAN with a novel mini-discriminator and a direct approach that trains a neural network using a novel loss function. The performance of both methods is demonstrated on image reconstruction in ultra low dose 3D helical CT. We compute the posterior mean and standard deviation of the 3D images followed by a hypothesis test to assess whether a \u0026lsquo;dark spot\u0026rsquo; in the liver of a cancer stricken patient is present. Both methods are computationally efficient and our evaluation shows very promising performance that clearly supports the claim that Bayesian inversion is usable for 3D imaging in time critical applications.\nPDF · Preprint\n","date":"16 November 2018","permalink":"https://jonasadler.com/publications/deep_bayesian/","section":"Publications","summary":"We use deep learning to perform bayesian inversion.","title":"Deep Bayesian Inversion"},{"content":"University of Bath · Bath, England\nDeep learning has shown tremendous success in solving various tasks in several fields of science such as in image and natural language processing. Recently, it has also been applied to solve inverse problems and empirical evidence in image reconstruction points to exponential improvements in both performance and run-time over classical approaches. In this talk we outline some of these recent developments.\\n In particular, we\u0026rsquo;ll introduce \u0026lsquo;Learned Iterative Reconstruction\u0026rsquo;, a method which relies on a fusion of model- and data-driven approaches for solving inverse problems.\\n We\u0026rsquo;ll also show how these methods can be extended to \u0026lsquo;Deep Bayesian Inversion\u0026rsquo;, a family of methods that allows us to perform uncertainty quantification using deep learning.\nPDF\n","date":"4 November 2018","permalink":"https://jonasadler.com/talks/2018unibath/","section":"Talks","summary":"University of Bath — Bath, England","title":"Invited talk: Deep Learning for Image Reconstruction"},{"content":"Chinese Academy of Sciences · Beijing, China\nDeep learning has shown tremendous success in solving various tasks in several fields of science such as in image and natural language processing. Recently, it has also been applied to solve inverse problems and empirical evidence in image reconstruction points to exponential improvements in both performance and run-time over classical approaches. In this talk we outline some of these recent developments.\nIn particular, we\u0026rsquo;ll introduce \u0026lsquo;Learned Iterative Reconstruction\u0026rsquo;, a method which relies on a fusion of model- and data-driven approaches for solving inverse problems.\nWe\u0026rsquo;ll also show how these methods can be extended to \u0026lsquo;Deep Bayesian Inversion\u0026rsquo;, a family of methods that allows us to perform uncertainty quantification using deep learning.\n","date":"19 October 2018","permalink":"https://jonasadler.com/talks/2018icmsec/","section":"Talks","summary":"Chinese Academy of Sciences — Beijing, China","title":"Invited talk: Deep Learning for Image Reconstruction"},{"content":"Jonas Adler, Sebastian Lunz, Olivier Verdier, Carola-Bibiane Schönlieb, Ozan Öktem · arXiv\nThe paper considers the problem of performing a task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and task as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a differentiable parametrization of the family of estimators for both steps. These networks are combined and jointly trained against suitable supervised training data in order to minimize a joint differentiable loss function, resulting in an end-to-end task adapted reconstruction method. The suggested framework is generic, yet adaptable, with a plug-and-play structure for adjusting both the inverse problem and the task at hand. More precisely, the data model (forward operator and statistical model of the noise) associated with the inverse problem is exchangeable, eg, by using neural network architecture given by a learned iterative method. Furthermore, any task that is encodable as a trainable neural network can be used. The approach is demonstrated on joint tomographic image reconstruction, classification and joint tomographic image reconstruction segmentation.\nPDF · Preprint\n","date":"27 August 2018","permalink":"https://jonasadler.com/publications/task_adapted_recon/","section":"Publications","summary":"Task adapted reconstruction for inverse problems.","title":"Task adapted reconstruction for inverse problems"},{"content":"Nikita Moriakov, Koen Michielsen, Jonas Adler, Ritse Mann, Ioannis Sechopoulos, Jonas Teuwen · arXiv\nDigital breast tomosynthesis is rapidly replacing digital mammography as the basic x-ray technique for evaluation of the breasts. However, the sparse sampling and limited angular range gives rise to different artifacts, which manufacturers try to solve in several ways. In this study we propose an extension of the Learned Primal-Dual algorithm for digital breast tomosynthesis. The Learned Primal-Dual algorithm is a deep neural network consisting of several `reconstruction blocks\u0026rsquo;, which take in raw sinogram data as the initial input, perform a forward and a backward pass by taking projections and back-projections, and use a convolutional neural network to produce an intermediate reconstruction result which is then improved further by the successive reconstruction block. We extend the architecture by providing breast thickness measurements as a mask to the neural network and allow it to learn how to use this thickness mask. We have trained the algorithm on digital phantoms and the corresponding noise-free/noisy projections, and then tested the algorithm on digital phantoms for varying level of noise. Reconstruction performance of the algorithms was compared visually, using MSE loss and Structural Similarity Index. Results indicate that the proposed algorithm outperforms the baseline iterative reconstruction algorithm in terms of reconstruction quality for both breast edges and internal structures and is robust to noise.\nPDF · Preprint\n","date":"14 August 2018","permalink":"https://jonasadler.com/publications/deep_learning_digital_breast_tomosynthesis/","section":"Publications","summary":"Learned Primal-Dual for DBT","title":"Deep Learning Framework for Digital Breast Tomosynthesis Reconstruction"},{"content":"Sebastian Banert, Axel Ringh, Jonas Adler, Johan Karlsson, Ozan Öktem · arXiv\nIn this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first specify a class of optimization algorithms using a generic iterative scheme involving only linear operations and applications of proximal operators. This scheme contains many modern primal-dual first-order solvers like the Douglas-Rachford and hybrid gradient methods as special cases. Moreover, we show convergence to an optimal point for a new method which also belongs to this class. Next, we interpret the generic scheme as a neural network and use unsupervised training to learn the best set of parameters for a specific class of objective functions while imposing a fixed number of iterations. In contrast to other approaches of \u0026rsquo;learning to optimize\u0026rsquo;, we present an approach which learns parameters only in the set of convergent schemes. As use cases, we consider optimization problems arising in tomographic reconstruction and image deconvolution, and in particular a family of total variation regularization problems.\nPDF · Preprint · Code\n","date":"2 August 2018","permalink":"https://jonasadler.com/publications/data_driven_nonsmooth_optimization/","section":"Publications","summary":"We show how learning can be applied to proximal Primal-Dual schemes while still guaranteeing convergence.","title":"Data-driven Nonsmooth Optimization"},{"content":"Zhichao Zhong, Willem Jan Palenstijn, Jonas Adler, K. Joost Batenburg · Ultramicroscopy\nEnergy-dispersive X-ray spectroscopy (EDS) tomography is an advanced technique to characterize compositional information for nanostructures in three dimensions (3D). However, the application is hindered by the poor image quality caused by the low signal-to-noise ratios and the limited number of tilts, which are fundamentally limited by the insufficient number of X-ray counts. In this paper, we explore how to make accurate EDS reconstructions from such data. We propose to augment EDS tomography by joining with it a more accurate high-angle annular dark-field STEM (HAADF-STEM) tomographic reconstruction, for which usually a larger number of tilt images are feasible. This augmentation is realized through total nuclear variation (TNV) regularization, which encourages the joint EDS and HAADF reconstructions to have not only sparse gradients but also common edges and parallel (or antiparallel) gradients. Our experiments show that reconstruction images are more accurate compared to the non-regularized and the total variation regularized reconstructions, even when the number of tilts is small or the X-ray counts are low.\n","date":"1 August 2018","permalink":"https://jonasadler.com/publications/eds_tomo_recon/","section":"Publications","summary":"Ultramicroscopy","title":"EDS tomographic reconstruction regularized by total nuclear variation joined with HAADF-STEM tomography"},{"content":"Jonas Adler, Sebastian Lunz · NIPS\nWasserstein Generative Adversarial Networks (WGANs) can be used to generate realistic samples from complicated image distributions. The Wasserstein metric used in WGANs is based on a notion of distance between individual images, which induces a notion of distance between probability distributions of images. So far the community has considered ℓ2 as the underlying distance. We generalize the theory of WGAN with gradient penalty to Banach spaces, allowing practitioners to select the features to emphasize in the generator. We further discuss the effect of some particular choices of underlying norms, focusing on Sobolev norms. Finally, we demonstrate the impact of the choice of norm on model performance and show state-of-the-art inception scores for non-progressive growing GANs on CIFAR-10.\nPDF · Preprint · Code\n","date":"18 June 2018","permalink":"https://jonasadler.com/publications/bwgan/","section":"Publications","summary":"Generalizing Wasserstein GAN with gradient penalty to Banach Spaces.","title":"Banach Wasserstein GAN"},{"content":"SIAM conference on imaging science · Bologna, Italy\n","date":"8 June 2018","permalink":"https://jonasadler.com/talks/2018siam_ct/","section":"Talks","summary":"SIAM conference on imaging science — Bologna, Italy","title":"Invited talk: Learned Iterative Reconstruction for CT"},{"content":"SIAM conference on imaging science · Bologna, Italy\n","date":"8 June 2018","permalink":"https://jonasadler.com/talks/2018siam_odl/","section":"Talks","summary":"SIAM conference on imaging science — Bologna, Italy","title":"Invited talk: Learning to solve inverse problems with ODL"},{"content":"High Performance Scientific Computing · Hanoi, Vietnam\n","date":"23 March 2018","permalink":"https://jonasadler.com/talks/2018hpsc/","section":"Talks","summary":"High Performance Scientific Computing — Hanoi, Vietnam","title":"Invited talk: What Can We Expect? Computable Upper Bounds to Machine Learning in Inverse Problems Using MCMC"},{"content":"Swedish Symposium on Image Analysis · Stockholm, Sweden\n","date":"9 March 2018","permalink":"https://jonasadler.com/talks/2018ssba/","section":"Talks","summary":"Swedish Symposium on Image Analysis — Stockholm, Sweden","title":"Contributed talk: Learned iterative reconstruction"},{"content":"Neural Information Processing Systems · Los Angeles, USA\n","date":"9 December 2017","permalink":"https://jonasadler.com/talks/2017nips/","section":"Talks","summary":"Neural Information Processing Systems — Los Angeles, USA","title":"Poster: Learning to solve inverse problems using Wasserstein Loss"},{"content":"Generative models, parameter learning and sparsity · Cambridge, UK\n","date":"31 October 2017","permalink":"https://jonasadler.com/talks/2017generative_models_parameter/","section":"Talks","summary":"Generative models, parameter learning and sparsity — Cambridge, UK","title":"Contributed talk: Learned forward operators: Variational regularization for black-box models"},{"content":"Jonas Adler, Axel Ringh, Ozan Öktem, Johan Karlsson · NIPS workshop in optimal transport\nWe propose using the Wasserstein loss for training in inverse problems. In particular, we consider a learned primal-dual reconstruction scheme for ill-posed inverse problems using the Wasserstein distance as loss function in the learning. This is motivated by miss-alignments in training data, which when using standard mean squared error loss could severely degrade reconstruction quality. We prove that training with the Wasserstein loss gives a reconstruction operator that correctly compensates for miss-alignments in certain cases, whereas training with the mean squared error gives a smeared reconstruction. Moreover, we demonstrate these effects by training a reconstruction algorithm using both mean squared error and optimal transport loss for a problem in computerized tomography.\nPDF · Preprint · Code\n","date":"30 October 2017","permalink":"https://jonasadler.com/publications/learning_inverse_wasserstein/","section":"Publications","summary":"Compute the image-space Wasserstein loss using Sinkhorn iterations and use it for learning to solve inverse problems.","title":"Learning to solve inverse problems using Wasserstein loss"},{"content":"Variational Methods Meet Machine Learning · Cambridge, UK\n","date":"18 September 2017","permalink":"https://jonasadler.com/talks/2017variationalmethodsml/","section":"Talks","summary":"Variational Methods Meet Machine Learning — Cambridge, UK","title":"Invited talk: Learned iterative reconstruction schemes, theory and practice"},{"content":"Andreas Hauptmann, Felix Lucka, Marta Betcke, Nam Huynh, Jonas Adler, Ben Cox, Paul Beard, Sebastien Ourselin, Simon Arridge · IEEE - Transactions on Medical Imaging\nRecent advances in deep learning for tomographic reconstructions have shown great potential to create accurate and high quality images with a considerable speed-up. In this work we present a deep neural network that is specifically designed to provide high resolution 3D images from restricted photoacoustic measurements. The network is designed to represent an iterative scheme and incorporates gradient information of the data fit to compensate for limited view artefacts. Due to the high complexity of the photoacoustic forward operator, we separate training and computation of the gradient information. A suitable prior for the desired image structures is learned as part of the training. The resulting network is trained and tested on a set of segmented vessels from lung CT scans and then applied to in-vivo photoacoustic measurement data.\nPDF · Preprint · Code\n","date":"31 August 2017","permalink":"https://jonasadler.com/publications/model-based-learning-photoacoustic/","section":"Publications","summary":"Using gradient boosting we scale learned iterative reconstruction to a very large scale inverse problem.","title":"Model based learning for accelerated, limited-view 3D photoacoustic tomography"},{"content":"An introduction to some Machine Learning methods for image reconstruction.\nSuppose that you were given a blurred image:\nThen you could probably tell that it was generated from this image:\nBut what about if you instead were given indirect observations of the image, such as the line integrals taken over some set of lines:\nWould you still tell it came from the image above? Probably not. But we will demonstrate how we can teach a computer to do it.\nIntroduction #These problems are called the deblurring and radon inversion problems respectively and are both examples of inverse problems. In inverse problem we seek to find (reconstruct) some parameter given indirect observations. Mathematically this can be stated as finding some parameter $f$ that are connected to measured data $g$ through a known forward model $\\mathcal{T}$:\n$$ g = \\mathcal{T}(f) + noise $$\nSeveral classical image processing problems such as denoising, inpainting, deblurring and superresolution can be cast into this form, but it is also a good model for more complicated problems such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT).\nMachine learning for inverse problems #If the forward model is local, such as in deblurring where each datapoint is affected only by a small neighborhood of the image, finding the unknown parameter from data using machine learning is at least on a conceptual level straightforward. Simply use any standard convolutional neural network and known pairs $(g, f_{\\text{true}})$ and train the neural network to map $g$ to $f_{\\text{true}}$.\nBut what happens when $\\mathcal{T}$ is more complicated? In Computed Tomography, the parameter $f$ is an image of the interior of the patient and the forward model is given by the Radon Transform which computes all line integrals of the image:\n$$ \\mathcal{T}(f)(\\ell) = \\int_{\\ell} f(x) dx \\quad \\ell \\text{ is a line in } \\mathbb{R}^2 $$\nIf we compare this to the deblurring problem, we see that while the deblurring problem has a local relationship between parameter and data, the radon transform has a global relationship. Here two points on opposite sides of the image can still influence the same datapoint as long as they lie on the same line. And since the forward model is global, so is the inverse problem. This means that classical local machine-learning approaches based on convolutions will not work.\nOne way to solve this would be to use fully-connected layers instead of convolutions, but this quickly becomes infeasible. For example in the example above a single fully connected layer between data and signal would require a staggering $2.6 \\cdot 10^{11}$ weights. Storing these in single precision would require one terabyte of memory. For a single layer.\nThis is obviously not a workable solution.\nLearned denoisers #One of the most obvious solutions to this is to somehow recast the problem to an image-to-image problem. The most popular way of doing this is to first perform some initial (non machine-learning) reconstruction, e.g. Filtered Back-Projection (FBP):\nOnce this is done we can use any standard machine-learning approach to \u0026ldquo;denoise\u0026rdquo; the initial reconstruction by training a neural network to take the initial reconstruction as data and return the ground truth.\nSeveral groups have done this and the results are in fact quite remarkable, outperforming previous state of the art methods:\nHowever, the method leaves a sour taste. Sure the images certainly look better, but the only input was the initial reconstruction, so could it truly show anything that wasn\u0026rsquo;t already there?\nLearned Primal-Dual #This observation leads to a painful conclusion: in order to obtain a reconstruction with more information than current reconstruction methods, we need to work directly from raw data. But as we noted above, fully learning how to do this is practically impossible.\nThe solution is to take a middle way, to incorporate enough a-priori information to make the problem tractable and then learn the rest.\nThe most powerful prior information we have is the forward operator $\\mathcal{T}$, but it only maps images to data. How would we go from data to images? One answer is to use the adjoint operator $\\mathcal{T}^*$.\nThe idea of our proposed solution (called the Learned Primal-Dual algorithm) is to use these operators alternatively. First we use a convolutional neural network to update the data (a so called dual step), then apply $\\mathcal{T}^*$ and use the result as input to another neural network which updates the reconstruction (the primal step), then apply $\\mathcal{T}$ and use it as input to a neural network that updates the data, and so on. This is iteratively performed a few times (10 in our experiments), at which point we have a final reconstruction.\nWe train end-to-end with raw measured data as input and the reconstruction as output, without any initial reconstruction or other external information.\nThe good thing about this is that we separate the global aspect of the problem into the forward model and its adjoint and only need to learn the local aspects. The bad thing is that to train the network we need to perform back-propagation through this neural network that among others contain 10 calls to the forward operator, 10 calls the the adjoint operator and 20 small neural networks in between. We did this using some magic with Operator Discretization Library ODL and TensorFlow.\nResults #If you ask me, the results look quite good and I especially appreciate that the method is able to avoid some artifacts (some examples shown with red arrows) that the denoiser wasn\u0026rsquo;t able:\nThe quantitative results are also quite good and we outperform learned denoising w.r.t both Peak Signal to Noise Ratio (PSNR) and structural similarity index (SSIM). The runtime is not too shabby either and we manage to do all of this using only $2%$ of the trainable parameters used in the denoiser.\nMethod PSNR (dB) SSIM Runtime (ms) Parameters FBP 33.65 0.83 423 1 Denoiser 41.92 0.94 463 $10^7$ Proposed 44.11 0.97 620 $2.4 \\cdot 10^5$ Read more #If you found this interesting you should read our article \u0026ldquo;Learned Primal-Dual Reconstruction\u0026rdquo; on arXiv which describes the method in depth and gives a broader overview of what others have done in this exciting field. You could also have a look on the source code.\n","date":"22 July 2017","permalink":"https://jonasadler.com/posts/learning_to_reconstruct/","section":"Posts","summary":"\u003cp\u003eAn introduction to some Machine Learning methods for image reconstruction.\u003c/p\u003e","title":"Learning to reconstruct"},{"content":"","date":null,"permalink":"https://jonasadler.com/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"Jonas Adler, Ozan Öktem · IEEE - Transactions on Medical Imaging\nWe propose the Learned Primal-Dual algorithm for tomographic reconstruction. The algorithm accounts for a (possibly non-linear) forward operator in a deep neural network by unrolling a proximal primal-dual optimization method, but where the proximal operators have been replaced with convolutional neural networks. The algorithm is trained end-to-end, working directly from raw measured data and it does not depend on any initial reconstruction such as FBP. We compare performance of the proposed method on low dose CT reconstruction against FBP, TV, and deep learning based post-processing of FBP. For the Shepp-Logan phantom we obtain \u0026gt;6dB PSNR improvement against all compared methods. For human phantoms the corresponding improvement is 6.6dB over TV and 2.2dB over learned post-processing along with a substantial improvement in the SSIM. Finally, our algorithm involves only ten forward-back-projection computations, making the method feasible for time critical clinical applications.\nPDF · Preprint · Code\n","date":"5 July 2017","permalink":"https://jonasadler.com/publications/learned-primal-dual/","section":"Publications","summary":"By learning in both reconstruction and data domain, we can improve image reconstruction.","title":"Learned Primal-Dual Reconstruction"},{"content":"Jonas Adler, Gregory J. Bootsma, Håkan Nordström, Markus Eriksson · Fully3D\nSpectral CT allows reconstructing a set of materialselective basis images which can be used for material quantification. These basis images can be reconstructed independently of each other or treated as a joint reconstruction problem. In this work, we investigate the effect of two ways of introducing coupling between the basis images: using an anti-correlated noise model and regularizing the basis images with a joint prior. We simulate imaging of a FORBILD Head phantom with an ideal photon-counting detector and reconstruct the resulting basis sinograms with and without these two kinds of coupling. The results show that the anti-correlated noise model gives better spatial resolution than the uncorrelated noise model at the same noise level, but also introduces artifacts. If anti-correlations are introduced also in the prior, these artifacts are reduced and the resolution is improved further.\n","date":"1 June 2017","permalink":"https://jonasadler.com/publications/gpumci/","section":"Publications","summary":"Fully3D","title":"GPUMCI: a flexible platform for x-ray imaging on the GPU"},{"content":"Applied Inverse Problems · Hangzhou, China\n","date":"31 May 2017","permalink":"https://jonasadler.com/talks/2017aip/","section":"Talks","summary":"Applied Inverse Problems — Hangzhou, China","title":"Invited talk: Using deep learning to reconstruct multi-modal images - A primal dual scheme with examples in PET-MRI"},{"content":"Jonas Adler, Ozan Öktem · Inverse Problems\nWe propose a partially learned approach for the solution of ill-posed inverse problems with not necessarily linear forward operators. The method builds on ideas from classical regularisation theory and recent advances in deep learning to perform learning while making use of prior information about the inverse problem encoded in the forward operator, noise model and a regularising functional. The method results in a gradient-like iterative scheme, where the \u0026lsquo;gradient\u0026rsquo; component is learned using a convolutional network that includes the gradients of the data discrepancy and regulariser as input in each iteration.\nWe present results of such a partially learned gradient scheme on a non-linear tomographic inversion problem with simulated data from both the Sheep-Logan phantom as well as a head CT. The outcome is compared against filtered backprojection and total variation reconstruction and the proposed method provides a 5.4 dB PSNR improvement over the total variation reconstruction while being significantly faster, giving reconstructions of $512 \\times 512$ pixel images in about 0.4 s using a single graphics processing unit (GPU).\nPDF · Preprint · Code\n","date":"13 April 2017","permalink":"https://jonasadler.com/publications/learning-to-solve/","section":"Publications","summary":"A deep learning scheme for solving inverse problems that incorporates knowledge about the data formation process, improving both speed and quality compared to classical methods.","title":"Solving ill-posed inverse problems using iterative deep neural networks"},{"content":"Awais Ashfaq, Jonas Adler · IFMBE Proceedings\n","date":"1 January 2017","permalink":"https://jonasadler.com/publications/a_modified_fuzzy_c_means/","section":"Publications","summary":"IFMBE Proceedings","title":"A modified fuzzy C means algorithm for shading correction in craniofacial CBCT images"},{"content":"Mats Persson, Jonas Adler · Fully3D\nSpectral CT allows reconstructing a set of materialselective basis images which can be used for material quantification. These basis images can be reconstructed independently of each other or treated as a joint reconstruction problem. In this work, we investigate the effect of two ways of introducing coupling between the basis images: using an anti-correlated noise model and regularizing the basis images with a joint prior. We simulate imaging of a FORBILD Head phantom with an ideal photon-counting detector and reconstruct the resulting basis sinograms with and without these two kinds of coupling. The results show that the anti-correlated noise model gives better spatial resolution than the uncorrelated noise model at the same noise level, but also introduces artifacts. If anti-correlations are introduced also in the prior, these artifacts are reduced and the resolution is improved further.\n","date":"1 January 2017","permalink":"https://jonasadler.com/publications/spectral_ct_anticorrelated/","section":"Publications","summary":"Fully3D","title":"Spectral CT reconstruction with anti-correlated noise model and joint prior"},{"content":"Instructions for installing and configuring Ubuntu 16.04 LTE on a PC with two GPUs.\nThe primary GPU will be used for computations and the secondary one will be used for displaying graphics. In this example, the computer is a HP Z640 with NVIDIA Titan-X Pascal as primary GPU and a NVIDIA Quadro K620 as secondary GPU.\nNote: if the computer already have a working installation of Ubuntu 16.04 with build-essentials installed you should (probably) not have to reinstall it, meaning that the first two steps can be skipped. Moreover, sometimes the installation procedure works also without removing the secondary GPU (see step 1 and step 22).\nTurn off computer, ensure only the primary GPU is connected to the motherboard. Boot from USB with Ubuntu 16.04 LTS installation, install a fresh copy of Ubuntu 16.04 LTS but keep Windows partitions intact, e.g. by choosing \u0026ldquo;Erase Ubuntu xx.yy and reinstall\u0026rdquo;\nInstall build essentials.\n$ sudo apt-get install build-essential Go to https://developer.nvidia.com/cuda-downloads and download the latest CUDA toolkit for Ubuntu 16.04 by selecting the following in the web-interface that is offered:\nOperating system: Linux Architecture: x86_64 Distribution: Ubuntu Version: 16.04 Installer Type: runfile (local) This will download the latest version, which in the instructions will be is version 8.0.44, so the file that will be downloaded is \u0026ldquo;cuda_8.0.44_linux.run\u0026rdquo;\nOpen up a terminal and extract the separate installers via:\n$ mkdir ~/Downloads/nvidia_installers $ cd ~/Downloads $ sudo chmod a+x ./cuda_8.0.44_linux.run $ ./cuda_8.0.44_linux.run -extract=~/Downloads/nvidia_installers Completely uninstall anything in the ubuntu repositories with nvidia-*. I used synaptic and did a purge, AKA completely uninstall programs and configuration.\n$ sudo apt-get --purge remove nvidia-* No need to create an xorg.conf file. If you have one, remove it (assuming you have a fresh OS install).\n$ sudo rm /etc/X11/xorg.conf Create the /etc/modprobe.d/blacklist-nouveau.conf file with the 2 following lines:\nblacklist nouveau options nouveau modeset=0 Then do a\n$ sudo update-initramfs -u Reboot computer. Nothing should have changed in loading up menu. You should be taken to the login screen. Once there type: Ctrl + Alt + F1, and login to your user. Keep the next commands handy in another machine since now you are in tty.\nIn tty:\ncd ~/Downloads/nvidia_installers sudo service lightdm stop The top line is a necessary step for installing the driver.\nNow type\nsudo ./NVIDIA-Linux-x86_64-367.48.run --no-opengl-files The file \u0026ldquo;NVIDIA-Linux-x86_64-367.48.run\u0026rdquo; should have been created by the extract command at step (4.) above. It is important to include the opengl flag in the above command. If you miss that, either you will get stuck in ?login loop? or your computer would boot with a black screen at all times.\nAt some point you will get a warning message about 32-bit support, answer ok.\nWhen prompted \u0026ldquo;The distribution-provided pre-installed script failed are you sure you want to continue\u0026rdquo;. Answer continue.\nIf prompted to over-write xorg.conf, answer yes.\nPick defaults for all other options.\nNow install the toolkit also\nsudo ./cuda-linux64-rel-8.0.44-21122537.run sudo ./cuda-samples-linux-8.0.44-21122537.run Both the run-files above are suppose to have been created by the extract command at step (4.) above.\nPick defaults to all questions.\nPick yes for symbolic links.\nSet Environment path variables in .bashrc:\nexport PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH Reload bash in order to apply these settings\n$ exec bash Verify the driver version:\n$ cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 367.48 Sat Sep 3 18:21:08 PDT 2016 GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.1) Check CUDA driver version:\n$ nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2016 NVIDIA Corporation Built on Sun_Sep__4_22:14:01_CDT_2016 Cuda compilation tools, release 8.0, V8.0.44 At this point you can switch the lightdm back on again by doing:\n$ sudo service lightdm start Shut down computer, insert secondary GPU which in this example is a Quadro K620 card. Keep the display connected to the primary GPU.\nBoot computer, log in.\nCheck that both GPU cards are recognized by the system, by typing\n$ nvidia-smi Mon Jan 9 14:47:10 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 367.48 Driver Version: 367.48 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A | | 34% 44C P8 17W / 250W | 129MiB / 12187MiB | 4% Default | |-------------------------------+----------------------+----------------------+ | 1 Quadro K620 Off | 0000:03:00.0 On | N/A | | 24% 41C P0 3W / 30W | 1MiB / 2000MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 3954 G /usr/lib/xorg/Xorg 128MiB | +-----------------------------------------------------------------------------+ Generate Nvidia X11 configuration file and activate multiple GPU option:\n$ sudo nvidia-xconfig -multigpu=on Set the secondary GPU as the default GPU device for displaying graphics.\nFind the PCI port of the secondary GPU by typing\n$ nvidia-smi -a In the output that follow, find the PCI bus related to the secondary GPU. Simply look for text in output where the \u0026ldquo;Product Name\u0026rdquo; matches the secondary GPU and read the PCI bus number.\nIn my current output is given below, the PCI bus is \u0026ldquo;3\u0026rdquo;:\n[...] GPU 0000:03:00.0 Product Name : Quadro K620 Product Brand : Quadro Display Mode : Enabled Display Active : Enabled Persistence Mode : Disabled Accounting Mode : Disabled Accounting Mode Buffer Size : 1920 Driver Model Current : N/A Pending : N/A Serial Number : 0324114080422 GPU UUID : GPU-4c631408-4129-9d5d-fbf3-0588bc1ab5cf Minor Number : 1 VBIOS Version : 82.07.4E.00.0E MultiGPU Board : No Board ID : 0x300 GPU Part Number : N/A Inforom Version Image Version : 2012.0504.00.03 OEM Object : 1.1 ECC Object : N/A Power Management Object : N/A GPU Operation Mode Current : N/A Pending : N/A GPU Virtualization Mode Virtualization mode : None PCI Bus : 0x03 Device : 0x00 Domain : 0x0000 Device Id : 0x13BB10DE Bus Id : 0000:03:00.0 [...] Update X11 configuration file.\n$ sudo pico /etc/X11/xorg.conf Find the section\nSection \u0026quot;Device\u0026quot; Identifier \u0026quot;Device0\u0026quot; Driver \u0026quot;nvidia\u0026quot; VendorName \u0026quot;NVIDIA Corporation\u0026quot; EndSection and replace with\nSection \u0026quot;Device\u0026quot; Identifier \u0026quot;Device0\u0026quot; Driver \u0026quot;nvidia\u0026quot; VendorName \u0026quot;NVIDIA Corporation\u0026quot; BusID \u0026quot;PCI:3:0:0\u0026quot; EndSection Here, \u0026ldquo;3\u0026rdquo; should match the PCI bus as determined above.\nShutdown computer. Switch display cable so that it is connected to the secondary GPU. Start computer.\nValidate that both GPUs are active and that the secondary GPU is used\n$ nvidia-smi Mon Jan 9 14:56:11 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 367.48 Driver Version: 367.48 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A | | 24% 41C P0 17W / 250W | 1MiB / 12187MiB | 0% Default | |-------------------------------+----------------------+----------------------+ | 1 Quadro K620 Off | 0000:03:00.0 On | N/A | | 34% 44C P8 3W / 30W | 129MiB / 2000MiB | 4% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 1 3954 G /usr/lib/xorg/Xorg 128MiB | +-----------------------------------------------------------------------------+ Acknowledgement #This guide largely follows Installing CUDA with Ubuntu, but has been updated for Ubuntu 16.04 and two GPUs. Ozan Öktem helped compile the notes.\n","date":"28 December 2016","permalink":"https://jonasadler.com/posts/install_cuda/","section":"Posts","summary":"\u003cp\u003eInstructions for installing and configuring Ubuntu 16.04 LTE on a PC with two GPUs.\u003c/p\u003e","title":"Dual GPU configuration in Ubuntu 16.04 and CUDA 8.0"},{"content":"IEEE Medical Imaging Conference · San Diego, USA\n","date":"15 November 2015","permalink":"https://jonasadler.com/talks/2015ieeemic/","section":"Talks","summary":"IEEE Medical Imaging Conference — San Diego, USA","title":"ODL: A python library for inverse problems"},{"content":"","date":null,"permalink":"https://jonasadler.com/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":null,"permalink":"https://jonasadler.com/tags/","section":"Tags","summary":"","title":"Tags"}]