Deep learning has shown tremendous success in solving various tasks in several fields of science such as in image and natural language processing. Recently, it has also been applied to solve inverse problems and empirical evidence in image reconstruction points to drastic improvements over classical approaches. A natural question that arrises is: how good can it actually get? We try to answer this by characterizing the optimal solution given various loss functions. In addition to this, we show that these upper bounds are empirically approximated quite well using deep neural networks.