Enterprise software evaluations often start out as spreadsheets but soon run into problems. Why does this happen? For the same reason that accounting systems don\u2019t run on spreadsheets: spreadsheets are too manual. They don\u2019t scale up to handle the many hundreds or thousands of requirements found in a typical enterprise software evaluation.\n\n\nThe scaling problem is familiar to people in growing companies. Systems that originally worked well begin to fail as they are scaled up. Likewise, spreadsheets that worked well when starting an evaluation struggle as the number of requirements grows.\n\n\nEvaluating enterprise software is a lot of work. Managing this work in a spreadsheet seems the obvious choice, but that is deceptive. What many people don\u2019t realize is that at the enterprise level, they will end up building a complete evaluation system in a spreadsheet, which is always far more work than expected. In this article, we look at the limitations of using spreadsheets to evaluate enterprise software and conclude with suggestions.\n\nLimited cell formatting\n\nTo select best-fit software you need well-written requirements. Formatting and text size limitations of spreadsheet cells are real problems. Also, Excel and Google spreadsheets limit you to one link per cell. In reality, you sometimes need more than one link:\n\n\nWhen writing requirements, rather than explaining acronyms and terms used, you may want to include inline links to external references like Wikipedia.\nWhen rating a product against a requirement, you might want to include links to pages in an online manual and links to videos explaining that feature.\n\n\nIn spreadsheets you can only have one link per cell; you can\u2019t add inline links to words or terms in the cell text. As a workaround, you can add extra columns to the spreadsheet but this can get unmanageable very quickly.\n\nLimited data structure\n\nRelational databases have the concept of normalized data, where each data item exists only once in the dataset. Denormalized data is where multiple copies of data items are used. The problem with denormalized data is that when a data item is changed, it must be found and manually updated in multiple places to keep everything consistent.\n\n\nThe limited data structure of spreadsheets means that multiple copies of data are inevitable. Keeping everything consistent is difficult if not impossible. This flat and non-relational data structure is a core problem with spreadsheets.\n\nDenormalized requirements\n\nSpreadsheets usually start with all products on one tab, but limitations quickly lead to a separate tab for each product being evaluated. When you consider that every requirement on an evaluation should appear on a master list, and again on a tab for each product being evaluated, the problem of manually updating multiple requirements starts to become apparent.\n\nRequirements in multiple groups\n\nTo help manage large numbers of requirements, they are always organized in groups. In our experience, around 15% of requirements belong in two or more groups. For example, the requirement:\n\nControl data deletion\n\nThe system should control who can delete data and what data they can delete. Where users can delete data, it should be possible to view the deleted data, or even restore that data if necessary.\n\n\nCould belong to the following requirement groups:\n\n\nCompliance Audit controls\nSecurity Access control\nUsability Data entry\n\n\nIf this requirement was on an evaluation that had 8 products, the requirement would appear on that evaluation 27 times: 3 times on each of the 8 product tabs, and 3 times on the master list of requirements.\n\n\nWhat is needed is a system that has normalized requirements: one requirement can be in multiple groups and on multiple products. Edit the requirement anywhere and it updates everywhere.\n\nGroup\u00a0 weights\n\nGroups of requirements need to have relative weights. In practice, those weights are adjusted towards the end of an evaluation when the requirements are better understood.\n\n\nGroup weights tend to be hard coded into the spreadsheet. When a group weight must be changed, it must be changed in multiple places. Again, manually keeping denormalized data consistent is difficult.\n\nMulti-user problems\n\nGoogles spreadsheets allow multiple people to edit the same spreadsheet simultaneously, a huge advantage because it prevents multiple versions of the spreadsheet from proliferating. Microsoft has introduced this same feature into Office365. However, enterprise evaluations are large, and in our experience, large online spreadsheets can be excruciatingly slow. The alternative, a local Excel spreadsheet, always leads to the problem of reconciling multiple versions of the same spreadsheet.\n\nProduct scoring\n\nThe whole idea behind evaluating multiple products is to score them relative to your particular requirements. This is the gap analysis. When doing the gap analysis on a spreadsheet you need to design a scoring system. This should take into account the weights of various requirement groups and the weights of the individual requirements in those groups.\n\n\nScoring is made more difficult by the lack of variables in formulas. The usual way to overcome this is by adding an extra column to hold the variable, and then hiding that column. Manually keeping the scoring formulas consistent across a large evaluation is mission impossible. Even if you do manage to make formulas consistent, there is no way to know those formulas are consistent.\n\nSearch\n\nEnterprise evaluations are large, and you always need to search requirements, for example, to see if a potential requirement already exists. A simple search is easy in a spreadsheet, but advanced searches need to be built. While this is not particularly difficult, the spreadsheet is starting to turn into a programming project.\n\nAudit trails\n\nWith spreadsheets, it is difficult to see who made changes. While you can track versions of a spreadsheet, there is no audit trail of individual changes to show who did what.\n\nError checking\n\nBecause spreadsheets are so manual, it is very easy for errors to creep in. All error checking must be manually coded, which is where the spreadsheet morphs into a fully blown programming project. Usually the person responsible for the software evaluation ends up doing this work, which makes the evaluation project take even longer.\n\nProgress tracking\n\nEnterprise evaluations are a lot of work that sometimes never seems to end. Spreadsheets have no tools built in to track progress, making these projects even more difficult to manage.\n\nConclusion\n\nThe total cost of ownership of enterprise software runs into many millions. When selecting that software your goal is to maximize the return on the investment. That means finding the software best matched to your organization\u2019s particular needs. While spreadsheets are tremendously useful for some kinds of problems, evaluating enterprise software does not play to their strengths. They are too manual, and simply don\u2019t scale up to large evaluations. Using spreadsheets for enterprise evaluations takes an already large project and adds another layer of work and risk. Your choices are:\n\n\nPlough through the work using a spreadsheet and hope that sheer effort identifies the best-fit software.\nRealize how much work there is, take shortcuts and hope for the best.\nFind an application specifically designed for evaluating enterprise software.\n\n\nIf you select the third option, you may consider requirements management systems. While they excel at requirements management, they are optimized for software development and lack the features needed to evaluate and compare software products. There are cloud based RFP or procurement systems. While very usable, they tend to be optimized for the RFP process. Since we could not find an application tailored for enterprise software evaluation & selection, we developed our own which we use in our consulting practice. To help others solve this same software selection problem we have made our application available at no cost.\n\n\nAny new enterprise software project is a significant opportunity for a company to improve. To make the most of this opportunity, the real needs of the company must be discovered. Then the software that best meets those needs must be identified. In addition to selecting the software, the evaluation process should also provide significant input to the implementation. Spreadsheets simply don't cut it anymore - it's time to use tools designed for the job.