Mess up internal search and you\u2019ll frustrate your\n employees. But mess up external search and you\u2019ll\n alienate your customers. No wonder that e-commerce company\n execs like Jeff Zwelling of YLighting bear down hard on this\n problem: Zwelling changed his website\u2019s search engine\n three times in the past four years, unhappy with the search\n results that his company\u2019s site was giving\n customers\u2014or rather wasn\u2019t giving them. Nothing\n changed until the fourth try in late 2006.Graeme McCracken, the COO of RB Search, a subsidiary of Reed\n Business charged with making the publisher\u2019s content\n available through the consolidated Zibb.com site, faced the\n same frustration three years ago. His search engine\n didn\u2019t give readers a complete, accurate picture of his\n company\u2019s many magazines and newsletters.Mired in the problems of external search, both companies\n found that the Google approach\u2014the one most commonly\n tried first\u2014doesn\u2019t always keep customers happy.\n E-commerce and media businesses have similar needs for external\n search: guided navigation and contextual search to help users\n quickly narrow down their desired results using categories,\n user profiles and other metadata. Even database-driven\n e-commerce sites must go beyond database content to handle\n vague searches like \u201cred lamp,\u201d says Zwelling,\n YLighting\u2019s president.External (keyword) search must help customers get to the\n same result as using the site\u2019s navigation, says Chris\n Cummings, CIO of online retailer eToys Direct.By contrast, internal search focuses on discovering data\n \u201chidden\u201d in documents, databases, and so forth.\n Google follows the internal search approach: Users typically\n want anything that answers their query, not a specific,\n repeatable result.E-commerce vendors and content publishers have come to these\n realizations early, says Tony Byrne, founder of the research\n firm CMS Watch, because the success of search relates directly\n to sales of goods and advertising. But other businesses can use\n search to im\u00adprove customer self-service (and reduce\n expensive calls and e-mails to customer support staff), he\n notes.Such efforts are rare today. \u201cThere\u2019s no revenue\n from better customer service, so it\u2019s hard to fund these\n projects,\u201d says Brian Babineau, a senior analyst at the\n consultancy Enterprise Strategy Group. But he expects savvy\n companies to follow the media and e-commerce firms\u2019\n examples to increase customer retention.\n It's All About Context\n \n Many search engines will give external users access to\n your website\u2019s content. But not all provide the\n ability to infer context from the content and then let\n an enterprise refine and manage that context.Leading players include Endeca Technologies, InQuira, Progress Software, SLI Systems, Visual Sciences and Vivisimo. All but InQuira and\n Vivisimo also offer search-based merchandising\n capabilities for e-commerce sites. SLI Systems\n provides its tools as a hosted service, while the\n other tools are designed to be deployed at the\n enterprise.Teragram provides a tool to\n create the metadata from which various search\n engines can access the context.Several companies can help you extend external\n search capabilities. For example, Baynote tracks users across the\n Web to build a profile of interests that a search\n engine can use invisibly to better target search\n results. And Nexidia offers search technology\n for audio and video content, using analysis of the\n audio to determine contextual matches to search\n terms.\n\n \n \n \n\n Solve the Context Problem\n When you embark on an external search project, it\u2019s\n important not to get overwhelmed by an early\n requirement\u2014classifying all the data to be searched. One\n of the hardest issues for RB Search\u2019s McCracken was\n bringing context into the search tool. He tried to tag the\n source material in the content management system to make the\n right information available to the search engine. But with 200\n million documents and new ones being created all the time, the\n RB staff could not tag all the content to provide the\n categories that a search engine would use to find appropriate\n content, suggest related results and deliver related\n promotions. In fact, McCracken realized that perhaps only 2\n percent of the content had been tagged, despite all the effort\n spent over a couple years. Worse, \u201cthe tags were not\n consistent\u201d among Reed\u2019s subsidiary com\u00adpanies,\n he says.So McCracken brought in a tool from Teragram that helped\n automate tagging of content after the fact, using a rules\n engine. Doing so meant creating the taxonomies and an\n ontological (conceptual categorization) dictionary of 210,000\n terms\u2014something that must be kept up to date by\n people\u2014but this made the tagging of the 200 million\n documents possible, he notes. McCracken then deployed Fast\n Search & Transfer, a search engine that provides the\n ability for search users to navigate through the categorized\n results derived from the tagged content.The key to this software-assisted classification, McCracken\n says: You can\u2019t depend completely on automation. Human\n experts must adjust the software\u2019s rules and results. But\n when the tools are properly tuned to a company\u2019s content,\n IT can then apply them to a vast quantity of documents, he\n says.The U.S. General Services Administration took a similar\n approach to making public documents from multiple federal,\n state, and local government organizations available via the\n USA.gov website. It used Vivisimo\u2019s clustering technology\n to contextually index the content from the multiple websites\n and Microsoft\u2019s MSN to provide the search engine and\n index. GSA staffers now hand-tune the index and ontology as\n needed, and can create their own indexes quickly when the need\n arises, such as pulling together all Hurricane\n Katrina\u2013related resources when the devastating storm\n struck in 2005, says John Murphy, director of USA.gov\n technologies.To keep the index and ontology relevant, you\u2019ll need\n to regularly analyze search queries and results to detect new\n user search patterns and expectations, says Ken Harris, CIO of\n natural products distributor Shaklee. He made this realization\n after replacing an old search engine with one from Visual\n Sciences (until recently known as WebSideStory) as part of a\n general Web modernization effort. The new tool came with\n analytics capability to help define relevance in results.\u201cWe then realized we didn\u2019t know internally what\n relevance was,\u201d he says. He quickly began to fill that\n gap, so the staff could tune the results to improve sales.\n\n Make the Sales Connection\n In e-commerce, the underlying product data is typically\n well-structured and tagged, so the need for additional context\n may not be as apparent. (The tagging effort is also easier for\n e-commerce firms than for media companies, notes CMS\n Watch\u2019s Byrne.)Most companies know to account for common misspellings by\n creating internal term maps, so for example, a customer looking\n for pendant lights will still find them if he types\n \u201cpendent\u201d in a search query, YLighting\u2019s\n Zwelling notes. (\u201cPendant\u201d is misspelled in nearly\n half of his site\u2019s searches.)And most companies know that databases may not be\n consistent, due to human error or differences in\n suppliers\u2019 own taxonomies, so additional effort is needed\n to also search for synonyms and to look across multiple fields\n for some terms, he says.But as Zwelling discovered, customers don\u2019t think in\n terms of just product specifications that match to product\n databases. And this requires more sophisticated work. For\n example, a query for \u201cred table lamp\u201d could miss\n lamps that come in a red finish but where the color choices are\n not called out in the database\u2019s color field or\n description. But a search engine such as the SLI Systems hosted\n search tool that he uses can detect all red lamps despite\n taxonomic differences, then let customers quickly sort them by\n room or material, he says.Sometimes there\u2019s a hidden need to adjust context. At\n Broder Bros., which sells shirts and other items that can later\n be customized with company logos, executives assumed that basic\n keyword search was sufficient, since the company sells to\n distributors who know the product codes or have a paper\n catalog. But an analysis of search patterns showed that about\n 15 percent of all searches were free-form: These people were\n essentially researching what might be available, says Mike\n Fabrico, VP of IT. Broder Bros.\u2019 search approach\n didn\u2019t serve that need\u2014and potentially lost sales.\n So the company replaced its search engine with one from\n Progress Software that could support contextual searches.Another tip: Don\u2019t overlook failed searches, says John\n Cortez, director of applications at Shaklee. He regularly\n monitors searches that result in no hits: This helps him\n identify new contextual mappings that would lead to appropriate\n results, and determine products that customers might want but\n aren\u2019t offered. Then he can give sales an indication of\n potential opportunities.\n\n Mind the Gaps\n When a search engine has the right context to find the right\n results, the next challenge is to present them usefully. Most\n modern search engines can filter results based on checkbox and\n menu selections, as well as attributes such as price or\n availability.But many merchants will want to go beyond that. After\n YLighting\u2019s Zwelling analyzed search histories, he\n noticed that the sales conversion rates for some items were\n lower than expected, even after a successful keyword search.\n Further usability studies explained the gap: Even if a search\n for \u201cred lamp\u201d turned up a lamp that met the needs\n of a customer, the image displayed might show the lamp in a\n different color. People reacted to the image rather than the\n text\u2014and didn\u2019t realize the displayed lamp was\n available in red.Zwelling then added images tagged by color, so the search\n engine could display the appropriate finish. Sales increased,\n and he attributes part of that to the search changes. (He\n declines to quantify the sales uptick, noting that it had\n multiple possible factors.) At Reed, search traffic increased\n 59 percent after the search engine retooling, and total traffic\n grew 19 percent, McCracken says.Unfortunately, at many organizations today, external search\n doesn\u2019t rise to the CIO\u2019s attention, says\n Accenture\u2019s Michael Kuhn, practice lead, Accenture\n Information Management Service, Europe, Africa and Latin\n America. \u201cYet it\u2019s a top priority for the\n user,\u201d he says. One result: \u201cThere is a lack of\n skills in the IT department on how to deal with search. They\n think of the search technology only, not of the metadata\n underlying it. And search is treated as an afterthought of a\n Web presence strategy,\u201d Kuhn adds. That\u2019s a\n mistake.Galen Gruman is a frequent contributor to CIO. E-mail\n him at email@example.com.