Semi-Coupled Dictionary Learning with Applications to Image Super-resolution and Photo-Sketch Synthesis

[pdf] [bibtex] [code] [data] [poster] [supple]

Shenlong Wang1,2     Lei Zhang2   Yan Liang1   Quan Pan1   

1 Northwestern Polytechnical University      2The Hong Kong Polytechnic University       

{csslwang, cslzhang}, {panquan, liangyan}


   In various computer vision applications, often we need to convert an image in one style into another style for better visualization, interpretation and recognition; for examples, up-convert a low resolution image to a high resolution one, and convert a face sketch into a photo for matching, etc. A semi-coupled dictionary learning (SCDL) model is proposed in this paper to solve such cross-style image synthesis problems. Under SCDL, a pair of dictionaries and a mapping function will be simultaneously learned. The dictionary pair can well characterize the structural domains of the two styles of images, while the mapping function can reveal the intrinsic relationship between the two styles' domains. In SCDL, the two dictionaries will not be fully coupled, and hence much flexibility can be given to the mapping function for a flexible conversion across styles. Moreover, clustering and image nonlocal redundancy are introduced to enhance the robustness of SCDL. The proposed SCDL model is applied to image interpolation and photo-sketch synthesis, and the experimental results validated its generality and effectiveness in cross-style image synthesis.

Image Style Transformation

Flowchart of the proposed semi-coupled dictionary learning (SCDL) based image cross-style synthesis

Experimental Results

Experimental results on image super-resolution (scaling factor: 2). From left to right: low resolution image, high resolution ground-truth, and reconstructed images by Bicubic, ScSR [1], SAI [4], SME [3] and the proposed SCDL method.


Experimental results on image super-resolution (scaling factor: 3). From left to right: low resolution image, high resolution ground-truth, and reconstructed images by Bicubic, ScSR [1] and the proposed SCDL method.

Instructions to use our code


Please feel free to use, modify our codes according to your own needs. Wish the following instructions may help.


To reproduce super-resolution experiment, please run Image_SR.m directly.


To reproduce dictionary learning procedures, please run Dict_Train directly.


To collect image patches from your own training data, please refer to collectPatches and conduct several modifications


To adopt our semi-coupled dictionary framework in your own tasks, please call the coupledDL function in our code.


The sparse coding algorithm used in this package is from SPAMS toolbox [7]. Please make sure that the toolbox has been successfully installed in your computer.


For further questions, please contact: shenlong dot wang at gmail dot com



J. Yang, J. Wright, T. Huang, and Y. Ma. Image super-resolution via sparse representation. Image Processing, IEEE Transactions on, 19(11):2861–2873, 2010.


X. Wang and X. Tang. Face photo-sketch synthesis and recognition. IEEE transactions on pattern analysis and ma-chine intelligence, pages 1955–1967, 2008.


S. Mallat and G. Yu. Super-resolution with sparse mix-ing estimators. Image Processing, IEEE Transactions on, 19(11):2889–2900, 2010.


X. Zhang and X. Wu. Image interpolation by adaptive 2-d autoregressive modeling and soft-decision estimation. Image Processing, IEEE Transactions on, 17(6):887–896, 2008.


W. T. Freeman, Thouis R. Jones and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, 2002


A. Efros and W. T. Freeman, Image quilting for texture synthesis and transfer, SIGGRAPH, 2001


J. Mairal, F. Bach, J. Ponce and G. Sapiro, "Online Dictionary Learning for Sparse Coding", in ICML 2009

Last update: Sept 28, 2012