Fixing the Search Engine

by Toby Ward — “The search engine sucks!” is one of the most common complaints I come across. Naturally, most organizations immediately blame the search engine. They should point the finger at themselves.
“The search engine sucks!” is one of the most common complaints I come across. Naturally, most organizations immediately blame the search engine. They should point the finger at themselves.

Five years ago I wrote the article The search isn't broken, we're broken (Part I: Search success depends on people and rules). Five years later, not much has changed.

While the search technology itself sometimes is the problem, this is rarely the issue. Search technology has advanced impressively in recent years and yet inaccurate and irrelevant search results continually defeat users performing search queries.

Though some search engines may be sub-par, the more likely problem is an absence of people processes and rules for managing information.

“People are lazy,” Cory Doctorow, a technologist who maintains the popular weblog Boing Boing, told me when I first talked to him 5 years ago. “People are remarkably cavalier about their information and how it is stored. This laziness is bottomless…”

One way of capitalizing on the potential of the search function to insert keywords as meta tags within the actual content pages. But this requires rules and a rulebook, otherwise known as the corporate taxonomy. A taxonomy is a set of rules, or dictionary, for classifying or cataloguing information – whether on the Internet, intranet or shared drives via a LAN or WAN (see Don’t forget to add the tax(onomy).

Meta tags, simply put, are the tags or data that describe the information contained on a page or site. Think of a meta tag as the tag on your shirt collar – it identifies the type of shirt and describes it with information about the materials and the manufacturer. Meta tags can be used to describe the type of data in terms of keywords, description, department, date, author, etc.

However, searching the intranet is fundamentally different than searching the Web:
  • Employee intranet queries are generally far more precise in nature than the average consumer Web search
  • Employees have to find information quickly to do their jobs – not finding the right information is not an option
  • The Internet doesn’t have a taxonomy; the intranet requires one

Autonomy has released an interesting ‘white paper’ (brochure) on 5 Differences Between Business Search and Consumer Search. Most of the paper is designed to get you to buy the Autonomy engine (Ultraseek) and therefore this paper requires a ‘grain of salt.’ Nonetheless, the Autonomy list of 5 differences underlines some important points:

  1. Return Role-based Results - The tasks for which employees use information vary widely, depending on their department and their role within their company.
  2. Provide Multiple Methods of Searching - Standards for search relevance is higher in business. Employees want a single, correct answer to their information request.
  3. Search All Corporate Information Repositories - Corporate information is spread across a host of specialized secure business applications, databases, content management repositories, email systems and Web servers—all of which require special interfaces.
  4. Support Multiple Languages and File Formats - Employees need to access business documents in any language and from a dizzying array of word processing, spreadsheet, presentation, graphic, multimedia, compression and encoding formats.
  5. Enforce Corporate Security Models and Compliance Policies - Access to corporate content must be securely managed in the face of a new matrix of government regulatory mandates and privacy concerns.

The last time I saw Autonomy at work I was impressed; but its expensive and not the answer to every organization’s problems. Though search features such as relevance ‘tuning’ (where search results and rankings can be ‘tuned’ to meet changing employee information demands and priorities) and manual keyword association certainly help any engine’s effectiveness.

The bottom line: search is more than technology. Effective information retrieval and knowledge management principally requires:

  • rules and defined processes (taxonomy and meta tagging)
  • employees who are not only willing to follow the rules but actively participate in sharing information and knowledge
  • effective supporting technology (search, content management, etc.)

Therefore, like most enterprise challenges, there is no silver bullet – and it certainly doesn’t come off the shelf. There is no quick fix for your search problems. It requires a lot of work and diligence and it starts with process and rules. Of course, if your content owners don’t follow the rules and prescribed process then your search engine will continue to suck.
Toby Ward is the President of Prescient Digital Media which specializes in digital workplace and intranet planning, design and implementation. Contact us directly if you need help with your intranet.