Project supervisors: Anadi Chaman and Ivan Dokmanić

Convolutional sparse coding (CSC) based methods have been shown to provide excellent performance on image reconstruction tasks like super-resolution, MRI/CT reconstruction, de-raining, etc [1,2,3]. These approaches generate sparse representatios for images using convolutional dictionaries whose parameters can be trained in a task-driven manner over a dataset.

While the performance of traditional CSC based methods was not as effective as deep neural networks, recently proposed dictionary learning approach called ISTA U-Net [3] showed comparable performance to the U-Net on various image reconstruction tasks. In addition, unlike U-Net, the ISTA-UNet offers an additional advantage of being interpretible from a signal processing perspective.

Figure 1: Schematic illustration of the classic U-Net (left panel) and the dictionary model of the ISTA U-Net.



Project proposal

The image reconstruction tasks over which U-Net and ISTA UNet are known to be comparable are all convolutional in nature, i.e. the input and output images can be related via a convolution. However, U-Net has been shown to perform well even on highly non-convolutional problems, for instance, XOR decryption [4].

In this project, we have the following goals.

  1. We would like to evaluate the performance of dictionary learning methods (especially ISTA U-Net) on such non-convolutional tasks and compare them against U-Net.
  2. We would investigate the factors that enable the performance of U-Net on such tasks and experimentally assess the limits of the network's performance. Similarly, we would attempt to find the reasons for any differences between U-Net and ISTA U-Net.


This project would have a significant coding component. The student should be proficient in training deep learning models in pytorch/tensorflow. Additional background in signal processing would be helpful, though not necessarily required.

Interested students are requested to contact Anadi Chaman.


[1] S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng and L. Zhang, "Convolutional Sparse Coding for Image Super-Resolution," 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1823-1831, doi: 10.1109/ICCV.2015.212.

[2] M. Li et al., "Video Rain Streak Removal by Multiscale Convolutional Sparse Coding," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6644-6653, doi: 10.1109/CVPR.2018.00695.

[3] T. Liu, A. Chaman, D. Belius and I. Dokmanić, "Learning Multiscale Convolutional Dictionaries for Image Reconstruction," in IEEE Transactions on Computational Imaging, vol. 8, pp. 425-437, 2022, doi: 10.1109/TCI.2022.3175309.

[4] Hauptmann, Andreas, and Jonas Adler. "On the unreasonable effectiveness of CNNs." arXiv preprint arXiv:2007.14745(2020).