Don’t rely on reviews when purchasing enterprise software

Reviews are often used when selecting enterprise software, but not many people consider their limitations. Let’s start by looking at two definitions:

Dictionary.com defines a review (noun) as:

• A general survey of something, especially in words; a report or account of something
• A critical article or report, as in a periodical, on a book, play, recital, or the like; critique; evaluation

An evaluation (noun) is defined as:

• An act or instance of evaluating or appraising

Notice that the definition of a “review” includes the word “evaluation.” So what is the difference between a review and an evaluation, and why does it matter when selecting software?

Reviews

In the context of buying enterprise software, a review is a critical examination done of one or more software products written to help organizations make a purchasing decision. Reviews are sometimes commissioned by software vendors, and those are usually biased in favor of that vendor. They are also written by analyst companies like Gartner, Forrester, IDC, etc. and sold. Occasionally they are written by knowledgeable users of a product who want to share their experience for the benefit of the community.

Reviews may include comparisons with other products, benchmarks, the strong and weak points of each product and often have graphical charts like Gartner’s famous Magic Quadrant. The most important thing about reviews is to realize they must be written for the “average” users of the products being reviewed. While they can be written for a subset audience, they are never written for just one organization.

One particularly useful feature of multiproduct reviews is that they can alert you to products you didn’t know you should be considering. A technique of finding relevant reviews on the Web is to search for the names of two or three software products in one search phrase. Reviews that include these products appear near the top of the search results, and these reviews sometimes include the names of products you don’t know about. When searching, remember to filter results by date; for example, a six-year-old review is much less useful than one that is six months old.

Evaluations

Like a review, an evaluation helps make a purchase decision. The primary purpose of an evaluation is to measure how well various potential software products fit the organization’s particular needs, where those needs are expressed as requirements.

For any given type of software, e.g. ERP, CRM, etc. we discovered that all organizations have essentially the same requirements. What makes each organization unique is how important those individual requirements are to them. For example, a financial institution will typically have much stricter security requirements that a retail organization.

With an evaluation, requirements may differ significantly from that of the average user of the software, which is why an evaluation is specific to an organization. Also, note that evaluations are far more thorough and detailed than reviews.

Conclusion

The primary difference between reviews and evaluations is that reviews are aimed outwards at multiple people. Software reviews are a very important source of information, especially in the early stages of a project when information is being gathered. However, because reviews are aimed at the average purchaser of the software, they are no more than a rough guide at best.

On the other hand, evaluations are aimed inwards, usually at a single purchase decision. Good evaluations are a detailed analysis that measures how well the evaluated software products meet an organization’s particular requirements.

Use reviews in the early stages of gathering information, but don’t make the mistake of thinking that because a product is recommended by one or more reviews you can’t go wrong. The TCO of enterprise software purchases often runs into tens of millions of dollars. Maximizing the return on that investment demands a thorough software evaluation. A bonus is that a thorough evaluation can also reduce software implementation risks.

This article is an updated version of an article that originally appeared on the Wayferry blog.