New Algorithms from UCSD Improve Automated Image Labeling

A Google image search for “tiger” yields many tiger photos, but also returns images of a tiger pear cactus stuck in a tire, a race car, Tiger Woods, the boxer Dick Tiger, Antarctica and many others. Why? Today’s large Internet search engines look for images using captions or other text linked to images rather than looking at what is actually in the picture.

Electrical engineers from UC San Diego are making progress on a different kind of image search engine—one that analyzes the images themselves, according to a university statement. This approach may be folded into next-generation image search engines for the Internet; and in the shorter term, it could be used to annotate and search commercial and private image collections.

“You might finally find all those unlabeled pictures of your kids playing soccer that are on your computer somewhere,” said Nuno Vasconcelos, a professor of electrical engineering at the UCSD Jacobs School of Engineering, and senior author of a paper in the March 2007 issue of the IEEE journal TPAMI, a paper coauthored by Gustavo Carneiro, a UCSD postdoctoral researcher now at Siemens Corporate Research, UCSD doctoral candidate Antoni Chan and Google researcher Pedro Moreno.

At the core of this Supervised Multiclass Labeling (SML) system is a set of simple yet powerful algorithms developed at UCSD. Once you train the system, you can set it loose on a database of unlabeled images. The system calculates the probability that various objects or “classes” it has been trained to recognize are present—and labels the images accordingly. After labeling, images can be retrieved via keyword searches. Accuracy of the UCSD system has outpaced that of other content-based image labeling and retrieval systems in the literature, says the university statement. The SML system also splits up images based on content—the historically difficult task of image segmentation. For example, the system can separate a landscape photo into mountain, sky and lake regions.

“Right now, Internet image search engines don’t use any image content analysis. They are highly scalable in terms of the number of images they can search but very constrained on the kinds of searches they can perform. Our semantic search system is not fully scalable yet, but if we’re clever, we will be able to work around this limitation. The future is bright,” said Vasconcelos.

The UCSD system uses a clever image-indexing technique that allows it to cover larger collections of images at a lower computational cost than was previously possible, claim researchers. While the current version would still choke on the Internet’s vast numbers of public images, there is room for improvement and many potential applications beyond the Internet, including the labeling of images in various private and commercial databases.