Rediscovering enterprise search

See also: Market report on enterprise search

The recent wave of mergers and acquisitions would seem to suggest that the enterprise search market is maturing as the number of players condenses.

With Oracle buying Endeca and HP taking Autonomy, it would seem there is little in the enterprise market to get nervous about.

But consider the fact that 2.5 quintillion bytes of data are sent from sensors, mobile devices, online transactions and social networks every day. CIOs must help make sense of these information sources.

Simon Price, European director at search vendor Recommind, says that IT departments must make people aware that effective enterprise search does not follow the same principles as web search.

“Enterprise search requires a much higher level of precision and recall than users are used to on the web. In web search, if users are unable to find their desired results within the first page, they tend to amend their search terms by adding more keywords,” says Price.

“But if employees don’t find the desired results at the top of an internal search, they presume the information is not available, or often blame the IT team,” he adds.

Any fault, says Whit Andrews, Gartner Group analyst, will always be with the content creators.

“They may not have developed the content that the users want. If the content exists, the question is whether the search engine can find it, and then whether it knows how to analyse it,” he says.

The next challenge for CIOs is whether their organisation’s query capture mechanism is effective.

Then they must consider whether results are displayed in a way that allows the user to participate in a conversation resulting in successful document/data location or analysis.

Erik de Muinck Keizer, the head of Google’s enterprise search division for EMEA, cites a recent McKinsey report (The Impact of Internet Technologies Search) which identified one trillion dollars worth of productivity value that search has contributed to the global economy.

By linking this to a report from IDC that says 39 per cent of an executive’s time is wasted on fruitless hunts for the right person or the right information in their own company, then, he argues, there are good grounds to say that search has a much bigger contribution to make to industry.

Given that the IDC report quantifies each knowledge worker as costing $200,000 a year (£128,000) (based on salary, training, maintenance and other factors) then, logically, each knowledge worker in your company is wasting $78,000 (£50,000) a year on fruitless searches.

The problem with enterprise search is its lack of usability, says Keizer.

This is something that Google can address because its priority has always been to simplify search.

“Simplicity drives usability,” he says.

The introduction of Google’s search appliances has created massive cost savings and productivity gains in banks, governments and manufacturers, he claims, by unifying the various systems into one cohesive whole.

“Our search appliances brought a level of visibility and confidence to end users that they hadn’t had before,” he says.

One of the reasons the old disciplines of search are not as effective is the sheer weight of content that users are creating, receiving and subsequently retaining.

A forgotten art

This wouldn’t be so bad if the information was tagged properly. But as more volumes are created, annotating this content in such a way as to make recalling it simple and straightforward is becoming a forgotten art.

There’s a psychological fault at play, says Matt Mullen, senior solutions consultant on semantic technologies for enterprise search software vendor OpenText.

“We still tend to think that document recovery, via search, is the real end-to-end process we are undertaking,” he says. Instead it is actually a small step in a larger workflow, which might include attempting to interrogate the knowledge base on what the enterprise collectively knows on any given subject.

The information in the aforementioned silos needs to be analysed and contextualised, Mullen explains.

“If you presume that an enterprise search instance is federating across all internal content silos, then by association, it has the potential to be able to produce enormous insight into the way in which an organisation is functioning at any given time,” says Mullen.

“Many organisations that we speak to still have fundamental problems finding what would seem to be fairly straightforward information about themselves,” says Mullen.

With so much of today’s new data coming from social networks, how should CIOs harness this unstructured information through search?

“On the face of it, social media just becomes another ingestion task for enterprise search,” says Mullen. But getting real analytical data from such content requires a great deal of thought beyond just pointing towards a source.

It begs a number of questions: how do we judge what is and is not an authoritative source? What is the size and make-up of the audience that reads that message? How do we judge the positive elements of messages without manually processing them all?

Given that social media works in real time, enterprise search tools will struggle to deal with  huge weight of content. The previous challenges that search systems faced, over precision and recall, are tiny by comparison.

“It would be easy to be overwhelmed by the weight of new sources. It’s why having a strategy for quickly annotating and reporting on this data is essential,” says Mullen.

The investments in search-powered applications – for the likes of Sharepoint, email, eDisclosure and records management – are all legacies that the CIO must sustain.

By providing an easy way to automatically and accurately access all relevant content, CIOs can immediately improve adoption of these applications, says Simon Price at Recommind.

At the same time, concept search, not uniformity of data, is the way forward. It could replace keyword search, which is over-inclusive and forces users to scroll through pages and pages of results. Concept-based search, meanwhile, allows users to fine-tune results quickly to pinpoint the data that is needed.

“Helping employees save time when looking for the information they need to do their jobs has an immediate effect on any organisation’s ROI, enabling companies to run lean with reduced resources,” says Price.

What’s in store?

In future enterprise search will be pushed into a context-embedded usage model, predicts Dan Matthews, CTO at enterprise applications developer IFS. Each major system and data source will have its own enterprise search box.

So you could expect one search box on the intranet, another in the mail program, and a third in the ERP system. These may all be connected to the same content index behind the scenes, but users will be unaware of this.

“Different indexing and search technologies are suitable for different types of data. A search engine that is brilliant at searching HTML content might be useless when searching a financial system filled with mainly numbers and dates,” Matthews explains.