Buyer beware: How auditing RFPs can help you make smarter enterprise software purchases

When buying enterprise software, organizations usually rely on vendors responding to RFPs (or RFIs or RFQs). However, as real world experience shows, responses to RFPs do not always reflect software reality. In all cases of RFP inaccuracies, the buyer pays the price in more ways than one. Why does this happen and what can you do about it? Two reasons are:

1. Poorly written RFP requirements. Salespeople always interpret unclear requirements in a way most favorable to their product.
2. Over Optimistic RFP Responses. Some salespeople are simply too aggressive when responding to RFPs.

The first problem can be remedied to some extent by the creator of the RFP. See this post on writing better requirements. The second problem is more insidious, remaining undetected until implementation or customer acceptance tests, when you discover the software does not truly meet the requirements. By then it is too late to do anything about it because the purchase has been made. While you can build some protections into the purchase contract, it is far better to avoid the problem in the first place.

At this point in the project, you may have done a thorough evaluation of potential products and selected the best-fit software for your organization’s particular needs. However, you are still relying on the vendor accurately responding to your RFP requirements. The way to uncover inaccurate and “over-optimistic” responses is to audit the RFP to verify the software does indeed perform as expected.

When aggressive sales people make “over-optimistic” RFP responses, the audit exposes what appeared to be the best-fit software for what it actually is. The true best-fit software can then be selected. The audit has caught the problem before making the purchase, and the organization avoided a potential software disaster.

Auditing an RFP

When initially entering RFP responses from vendors into the evaluation, mark each response to show the rating method, which documents the source of that rating. For example, since the RFP responses are typically by email, and any subsequent clarifications are by email or phone, we usually mark vendor responses as “Verbal or email.”

Ask the vendor for evidence that their software does indeed meet your requirements. Marketing materials are seldom detailed enough for RFP audits. Rather you need the following:

• User and admin documentation that describes features, and how to configure and use them
• Vendor demo videos showing how to use features
• An online account where you can login and test if features adequately satisfy your requirements

To audit an RFP, extract all “Showstopper” and “Critical” requirements where the vendor claims “Fully meets.” By definition, all these requirements have a fit score of 100 percent. See previous post on RFP scoring. For each requirement in the extract, examine the evidence to verify the software adequately meets it. If a requirement is inadequately met, reduce the rating appraisal from “Fully meets” to something lower, e.g. “Partly meets” or even “Does not meet.”

After auditing a requirement, update the product rating method to reflect this change, for example to “Audited” in the screenshot below. Add supporting evidence in the rating comments, for example, include a quote and a link to the source documentation. If the feature was tested by logging into the software, include a description of the test and the results.

Wayferry

The final step of the audit is to filter out everything except the audited product ratings. Recall that the fit score for the requirements in the extracted sample was 100 percent because all requirements are fully met. If you have audited a large enough sample of requirements and the fit score decreases significantly, you can assume the fit score for all requirements will decrease by a similar amount. We have seen products drop from ranking first place to third place or lower. On the other hand, if the fit score remains substantially the same after the audit, you can be confident that the RFP accurately reflects how well the software will meet your organization’s requirements.

Once a software product has passed the audit you can proceed with the purchase, confident that it matches your organization’s needs. You have substantially reduced project risks, and can take the next step forward in a successful enterprise software implementation.

The next post describes using product ratings to manage software implementation costs.