In Search of the Right Search Technology for Your Customers

Mess up internal search and you’ll frustrate your employees. But mess up external search and you’ll alienate your customers. No wonder that e-commerce company execs like Jeff Zwelling of YLighting bear down hard on this problem: Zwelling changed his website’s search engine three times in the past four years, unhappy with the search results that his company’s site was giving customers—or rather wasn’t giving them. Nothing changed until the fourth try in late 2006.

Graeme McCracken, the COO of RB Search, a subsidiary of Reed Business charged with making the publisher’s content available through the consolidated Zibb.com site, faced the same frustration three years ago. His search engine didn’t give readers a complete, accurate picture of his company’s many magazines and newsletters.

Mired in the problems of external search, both companies found that the Google approach—the one most commonly tried first—doesn’t always keep customers happy. E-commerce and media businesses have similar needs for external search: guided navigation and contextual search to help users quickly narrow down their desired results using categories, user profiles and other metadata. Even database-driven e-commerce sites must go beyond database content to handle vague searches like “red lamp,” says Zwelling, YLighting’s president.

External (keyword) search must help customers get to the same result as using the site’s navigation, says Chris Cummings, CIO of online retailer eToys Direct.

By contrast, internal search focuses on discovering data “hidden” in documents, databases, and so forth. Google follows the internal search approach: Users typically want anything that answers their query, not a specific, repeatable result.

E-commerce vendors and content publishers have come to these realizations early, says Tony Byrne, founder of the research firm CMS Watch, because the success of search relates directly to sales of goods and advertising. But other businesses can use search to im­prove customer self-service (and reduce expensive calls and e-mails to customer support staff), he notes.

Such efforts are rare today. “There’s no revenue from better customer service, so it’s hard to fund these projects,” says Brian Babineau, a senior analyst at the consultancy Enterprise Strategy Group. But he expects savvy companies to follow the media and e-commerce firms’ examples to increase customer retention.

It’s All About Context

Many search engines will give external users access to your website’s content. But not all provide the ability to infer context from the content and then let an enterprise refine and manage that context.

Leading players include Endeca Technologies, InQuira, Progress Software, SLI Systems, Visual Sciences and Vivisimo. All but InQuira and Vivisimo also offer search-based merchandising capabilities for e-commerce sites. SLI Systems provides its tools as a hosted service, while the other tools are designed to be deployed at the enterprise.

Teragram provides a tool to create the metadata from which various search engines can access the context.

Several companies can help you extend external search capabilities. For example, Baynote tracks users across the Web to build a profile of interests that a search engine can use invisibly to better target search results. And Nexidia offers search technology for audio and video content, using analysis of the audio to determine contextual matches to search terms.

Solve the Context Problem

When you embark on an external search project, it’s important not to get overwhelmed by an early requirement—classifying all the data to be searched. One of the hardest issues for RB Search’s McCracken was bringing context into the search tool. He tried to tag the source material in the content management system to make the right information available to the search engine. But with 200 million documents and new ones being created all the time, the RB staff could not tag all the content to provide the categories that a search engine would use to find appropriate content, suggest related results and deliver related promotions. In fact, McCracken realized that perhaps only 2 percent of the content had been tagged, despite all the effort spent over a couple years. Worse, “the tags were not consistent” among Reed’s subsidiary com­panies, he says.

So McCracken brought in a tool from Teragram that helped automate tagging of content after the fact, using a rules engine. Doing so meant creating the taxonomies and an ontological (conceptual categorization) dictionary of 210,000 terms—something that must be kept up to date by people—but this made the tagging of the 200 million documents possible, he notes. McCracken then deployed Fast Search & Transfer, a search engine that provides the ability for search users to navigate through the categorized results derived from the tagged content.

The key to this software-assisted classification, McCracken says: You can’t depend completely on automation. Human experts must adjust the software’s rules and results. But when the tools are properly tuned to a company’s content, IT can then apply them to a vast quantity of documents, he says.

The U.S. General Services Administration took a similar approach to making public documents from multiple federal, state, and local government organizations available via the USA.gov website. It used Vivisimo’s clustering technology to contextually index the content from the multiple websites and Microsoft’s MSN to provide the search engine and index. GSA staffers now hand-tune the index and ontology as needed, and can create their own indexes quickly when the need arises, such as pulling together all Hurricane Katrina–related resources when the devastating storm struck in 2005, says John Murphy, director of USA.gov technologies.

To keep the index and ontology relevant, you’ll need to regularly analyze search queries and results to detect new user search patterns and expectations, says Ken Harris, CIO of natural products distributor Shaklee. He made this realization after replacing an old search engine with one from Visual Sciences (until recently known as WebSideStory) as part of a general Web modernization effort. The new tool came with analytics capability to help define relevance in results.

“We then realized we didn’t know internally what relevance was,” he says. He quickly began to fill that gap, so the staff could tune the results to improve sales.

Make the Sales Connection

In e-commerce, the underlying product data is typically well-structured and tagged, so the need for additional context may not be as apparent. (The tagging effort is also easier for e-commerce firms than for media companies, notes CMS Watch’s Byrne.)

Most companies know to account for common misspellings by creating internal term maps, so for example, a customer looking for pendant lights will still find them if he types “pendent” in a search query, YLighting’s Zwelling notes. (“Pendant” is misspelled in nearly half of his site’s searches.)

And most companies know that databases may not be consistent, due to human error or differences in suppliers’ own taxonomies, so additional effort is needed to also search for synonyms and to look across multiple fields for some terms, he says.

But as Zwelling discovered, customers don’t think in terms of just product specifications that match to product databases. And this requires more sophisticated work. For example, a query for “red table lamp” could miss lamps that come in a red finish but where the color choices are not called out in the database’s color field or description. But a search engine such as the SLI Systems hosted search tool that he uses can detect all red lamps despite taxonomic differences, then let customers quickly sort them by room or material, he says.

Sometimes there’s a hidden need to adjust context. At Broder Bros., which sells shirts and other items that can later be customized with company logos, executives assumed that basic keyword search was sufficient, since the company sells to distributors who know the product codes or have a paper catalog. But an analysis of search patterns showed that about 15 percent of all searches were free-form: These people were essentially researching what might be available, says Mike Fabrico, VP of IT. Broder Bros.’ search approach didn’t serve that need—and potentially lost sales. So the company replaced its search engine with one from Progress Software that could support contextual searches.

Another tip: Don’t overlook failed searches, says John Cortez, director of applications at Shaklee. He regularly monitors searches that result in no hits: This helps him identify new contextual mappings that would lead to appropriate results, and determine products that customers might want but aren’t offered. Then he can give sales an indication of potential opportunities.

Mind the Gaps

When a search engine has the right context to find the right results, the next challenge is to present them usefully. Most modern search engines can filter results based on checkbox and menu selections, as well as attributes such as price or availability.

But many merchants will want to go beyond that. After YLighting’s Zwelling analyzed search histories, he noticed that the sales conversion rates for some items were lower than expected, even after a successful keyword search. Further usability studies explained the gap: Even if a search for “red lamp” turned up a lamp that met the needs of a customer, the image displayed might show the lamp in a different color. People reacted to the image rather than the text—and didn’t realize the displayed lamp was available in red.

Zwelling then added images tagged by color, so the search engine could display the appropriate finish. Sales increased, and he attributes part of that to the search changes. (He declines to quantify the sales uptick, noting that it had multiple possible factors.) At Reed, search traffic increased 59 percent after the search engine retooling, and total traffic grew 19 percent, McCracken says.

Unfortunately, at many organizations today, external search doesn’t rise to the CIO’s attention, says Accenture’s Michael Kuhn, practice lead, Accenture Information Management Service, Europe, Africa and Latin America. “Yet it’s a top priority for the user,” he says. One result: “There is a lack of skills in the IT department on how to deal with search. They think of the search technology only, not of the metadata underlying it. And search is treated as an afterthought of a Web presence strategy,” Kuhn adds. That’s a mistake.

Galen Gruman is a frequent contributor to CIO. E-mail him at ggruman@zangogroup.com.