Semi-Coupled
Dictionary Learning with Applications to Image Super-resolution and
Photo-Sketch Synthesis
[pdf] [bibtex] [code] [data] [poster] [supple]
Shenlong
Wang1,2 Lei
Zhang2 Yan Liang1 Quan Pan1
1 Northwestern Polytechnical
University 2The Hong Kong Polytechnic
University
{csslwang, cslzhang}@comp.polyu.edu.hk,
{panquan, liangyan}@nwpu.edu.cn
Abstract
In various computer vision
applications, often we need to convert an image in one style into another
style for better visualization, interpretation and recognition; for examples,
up-convert a low resolution image to a high resolution one, and convert a
face sketch into a photo for matching, etc. A semi-coupled dictionary
learning (SCDL) model is proposed in this paper to solve such cross-style
image synthesis problems. Under SCDL, a pair of dictionaries and a mapping
function will be simultaneously learned. The dictionary pair can well
characterize the structural domains of the two styles of images, while the
mapping function can reveal the intrinsic relationship between the two
styles' domains. In SCDL, the two dictionaries will not be fully coupled, and
hence much flexibility can be given to the mapping function for a flexible
conversion across styles. Moreover, clustering and image nonlocal redundancy
are introduced to enhance the robustness of SCDL. The proposed SCDL model is
applied to image interpolation and photo-sketch synthesis, and the
experimental results validated its generality and effectiveness in
cross-style image synthesis.
Image Style
Transformation
|
|
Flowchart
of the proposed semi-coupled dictionary learning (SCDL) based image
cross-style synthesis
|
Experimental
Results
|
|
Experimental
results on image super-resolution (scaling factor: 2). From left to right:
low resolution image, high resolution ground-truth, and reconstructed
images by Bicubic, ScSR
[1], SAI [4], SME [3] and the proposed SCDL method.
|
|
|
Experimental
results on image super-resolution (scaling factor: 3). From left to right:
low resolution image, high resolution ground-truth, and reconstructed
images by Bicubic, ScSR
[1] and the proposed SCDL method.
|
Instructions to
use our code
|
1.
|
|
Please
feel free to use, modify our codes according to your own needs. Wish the
following instructions may help.
|
|
(1)
|
To
reproduce super-resolution experiment, please run Image_SR.m
directly.
|
|
(2)
|
To
reproduce dictionary learning procedures, please run Dict_Train
directly.
|
|
(3)
|
To
collect image patches from your own training data, please refer to collectPatches and conduct several modifications
|
|
(4)
|
To
adopt our semi-coupled dictionary framework in your own tasks, please call
the coupledDL function in our code.
|
|
2.
|
|
The
sparse coding algorithm used in this package is from SPAMS toolbox [7].
Please make sure that the toolbox has been successfully installed in your
computer.
|
|
3.
|
|
For
further questions, please contact: shenlong dot wang at gmail dot com
|
References
|
[1]
|
J.
Yang, J. Wright, T. Huang, and Y. Ma. Image super-resolution via sparse
representation. Image Processing, IEEE Transactions on, 19(11):2861–2873,
2010.
|
|
[2]
|
X.
Wang and X. Tang. Face photo-sketch synthesis and recognition. IEEE
transactions on pattern analysis and ma-chine intelligence, pages
1955–1967, 2008.
|
|
[3]
|
S.
Mallat and G. Yu. Super-resolution with sparse
mix-ing estimators. Image Processing, IEEE
Transactions on, 19(11):2889–2900, 2010.
|
|
[4]
|
X.
Zhang and X. Wu. Image interpolation by adaptive 2-d autoregressive
modeling and soft-decision estimation. Image Processing, IEEE Transactions
on, 17(6):887–896, 2008.
|
|
[5]
|
W.
T. Freeman, Thouis R. Jones and Egon C. Pasztor,
Example-based super-resolution, IEEE Computer Graphics and Applications,
2002
|
|
[6]
|
A.
Efros and W. T. Freeman, Image quilting for
texture synthesis and transfer, SIGGRAPH, 2001
|
|
[7]
|
J.
Mairal, F. Bach, J. Ponce and G. Sapiro, "Online Dictionary Learning for Sparse
Coding", in ICML 2009
|
Last update: Sept 28, 2012
|