Information processing: Adding a touch of color

Creating a high-quality realistic color image from a grayscale picture can be challenging. Conventional methods typically require the user’s input, either by using a scribbling tool to color the image manually or by using a color transfer. Both options can result in poor colorization quality limited by the user’s degree of skill or the range of reference images available.

Alex Yong-Sang Chia at the A*STAR’s Institute for Infocomm Research and co-workers1 have now developed a computer program that utilizes the vast amount of imagery available on the internet to find suitable color matches for grayscale images. The program searches hundreds of thousands of online color images, cross-referencing their key features and objects in the foreground with those of grayscale pictures.

“We have developed a method that takes advantage of the plentiful supply of internet data to colorize gray photos,” Chia explains. “The user segments the image into separate major foreground objects and adds semantic labels naming these objects in the gray photo. Our program then scans the internet using these inputs for suitable object color matches.”

Given the vast amount of visual data available online, not all of the chosen images are useful. Once the initial color images have been found, the program then filters them to find the most realistic and suitable matches for the grayscale object inputs.

“Our method automatically detects and segments salient objects from an internet photo,” explains Chia. “It then exploits shape and appearance information of these objects to compute its relevance to the original grayscale image data.”

The grayscale image is then automatically colored using the information collected from internet-based images (pictured). Plausible colorization of images is vitally important, however, as the human eye can quickly distinguish between real and ‘false’ coloring. To this end, the user has the final say over the choice of colors. “The program generates several image colorizations and the user can pick the one that fits best from a graphical user interface,” explains Chia.

To demonstrate the capability of the program, Chia and his team showed a group of people their colored grayscale images alongside real color pictures, asking them to identify which ones had been colored artificially. “Our colored images were classed as ‘real’ up to 65% of the time,” says Chia. “Overall the colorization results are visually pleasing and perceptually meaningful to users.”

The researchers hope to expand the range of applications using this technology in the future. They envision that the technology may one day become so powerful that it could be used to generate realistic animations.

The A*STAR-affiliated researchers contributing to this research are from the Institute for Infocomm Research

Associated links
http://www.research.a-star.edu.sg/research/6508
Journal information
Chia, A. Y. S. et al. Semantic colorization with internet images. ACM Transactions on Graphics (TOG) – Proceedings of ACM SIGGRAPH Asia 2011 30, 156 (2012).