Mark Twain famously remarked that there are three kinds of lies: lies, damned lies, and statistics. Today, many CIOs feel the same way about metrics.\n\nMetrics are only as good as their source. \u201cToo often, technology companies pay consulting or analyst firms to create metrics based on the best characteristics of their offerings,\u201d says Judith Hurwitz, CEO of Hurwitz Strategies, an emerging technology consulting firm. \u201cTherefore, CIOs must be cautious about taking metrics at face value [and] leaders need to understand the data behind the metrics.\u201d\n\nMetrics interpretation is essentially a numbers game, and as with any numbers game, it\u2019s possible to win or lose. Here are seven ways IT leaders are often misled by key performance indicators (KPIs) and other critical business and IT metrics.\n\n1. Not considering the source\n\nWhen studying a metric, it\u2019s important to know who created it and the data source. Results may be based on a survey, for instance. If so, ask how many people were surveyed and the roles they played in their respective organizations. Check as well to see whether the metrics are based on a well-proven methodology. \u201cIt\u2019s important to understand the research and data behind the metrics,\u201d Hurwitz says.\n\nAlso consider the metric\u2019s purpose. Will it be used as a planning tool? If so, will it help determine a business strategy, a technology selection, or some other need? \u201cMetrics are only one tool for decision-making,\u201d Hurwitz notes. \u201cTherefore, approach metrics with skepticism.\u201d\n\n2. Failing to collaborate with front-line personnel\n\nBy now, most enterprises have reached data maturity. \u201cIf your company has data, you\u2019re definitely leveraging it and trying to use insights from analytics to drive positive business outcomes,\u201d says John Loury, president and CEO of Cause + Effect Strategy, a business intelligence consulting firm. \u201cIt\u2019s 2022, we\u2019re past the age of DRIP \u2014 data rich, insight poor.\u201d\n\nLoury believes that most organizations don\u2019t dig deep enough when communicating with the front-line business personnel who will ultimately use collected metrics to make decisions and drive actions. Before building analytics, he recommends collecting business requirements from all involved parties. This means distilling metrics down to the data points most relevant to drive outcomes, Loury notes. \u201cPrioritize what most directly impacts the business decision your user is trying to make.\u201d\n\nLoury advises building and honing communication skills to convey metrics-based insights to team members. \u201cModern CIOs and analytics leaders need to be adept at pulling together the key metrics that will drive the most impact for a team and presenting them in a way that makes sense to the user and will help guide their behavior,\u201d he says.\n\nLoury adds that it\u2019s also time for CIOs to task their teams with truly understanding their users and building them tailored, effective analytics solutions. \u201cThe days of data leaders and their teams scrambling to build something \u2014 anything \u2014 and ship it to business teams are behind us,\u201d he explains. \u201cWe\u2019re living with the results of those days, where teams are inundated with wall-to-wall dashboards that tell them everything and nothing.\u201d\n\n3. Overlooking the importance of ownership, involvement, and balance\n\nMetrics present an excellent opportunity for ownership and staff involvement, as well as continuous improvement and process control. \u201cThe key to correctly interpreting metrics is to engage your whole team and use the metrics to collectively improve processes,\u201d says Paul Gelter, coordinator of CIO services at business and technology consulting firm Centric Consulting.\n\nWhen evaluating metrics, Gelter believes it\u2019s essential to strike a balance between cost, quality, and service. Cost metrics, for example, could be tracked in completed tickets per individual, yet ticket quality could be degraded by rework\/repeated tickets. \u201cService could then be impacted by the response time, backlog, and uptime,\u201d he notes. It\u2019s all about obtaining an optimal balance.\n\n4. Chasing the wrong numbers\n\nTime really is money, so don\u2019t squander precious hours scrutinizing irrelevant metrics. Clearly identify all goals before deciding which metrics to study. In most cases, metrics that don\u2019t support or reflect future decision options are unnecessary and, worse yet, distracting and time-wasting.\n\nOnce the goal has been fully defined, allocate sufficient time to understanding the factors that cause individual metrics to fluctuate, suggests Alex Levin, co-founder of technology and design studio L+R. Next, investigate how individual metrics are tied to one another, and what\u2019s likely to happen during different stages within an initiative\u2019s or project\u2019s lifecycle that might directly affect the KPIs being tracked.\n\nMeanwhile, don\u2019t waste staff time by concealing or hoarding conclusions. Levin advises sharing study results with your team, ensuring that each individual can use metric-driven insights to improve performance and\/or outcomes.\n\n5. Going it alone\n\nMetrics research and study shouldn\u2019t be a solitary endeavor. Mike Capone, CEO of analytics and data integration platform developer Qlik, and a former CIO, recommends working with functional area owners at the outset to gather and apply valuable contextual details. \u201cThese inputs and relationships give the CIO and the IT team the right level of understanding of what\u2019s actually happening in the business ... to support operational goals in the short- and long-term,\u201d he explains. Capone also recommends building strong advisory partnerships with C-suite and other key enterprise leaders.\n\n6. Trusting the numbers too much\n\nA heathy dose of skepticism can keep you from being led down the path to faulty conclusions. Remember Twain\u2019s quip about statistics and lies. There\u2019s always the possibility that the collected data is itself flawed in some way.\n\nData can be flawed in many ways. The sample size could be too low, the time scale could be off, or whoever collected the data might have their own conclusion to promote. \u201cIt\u2019s vitally important to make sure you completely understand how the data is collected and what\u2019s included in the scope before you can make a determination on what it\u2019s telling you,\u201d says Brian Winters, CTO at ERP software developer ECI Software Solutions.\n\nIn fact, any metric can be misleading, especially if you don\u2019t have a good overall understanding of the data. \u201cSystem metrics can be particularly misleading because they often provide metrics for a very small part of a large, complex system,\u201d Winters notes. \u201cThat narrow view can easily lead you down a rabbit hole.\u201d\n\n7. Failing to see beyond the stats\n\nMetrics, while typically insightful and valuable, may not tell the entire story. In fact, taking any metric at face value can occasionally lead to utterly wrong conclusions. \u201cSometimes, you have to dig deeper with other, less obvious metrics, to determine what\u2019s really happening,\u201d explains Adi Gelvan, CEO and co-founder of database software developer Speedb.\n\nFor example, a high memory utilization level reading might imply that an application is overloading memory. \u201cBut something completely different may be at issue \u2014 perhaps a component that\u2019s not cleaning up the memory fast enough,\u201d Gelvan says. Further investigation can point to the real bottleneck, which may not be in the memory at all. \u201cFor example, if the storage engine cannot effectively dump the data to disks while I\/O consumption is high, memory will fill up fast and impact the performance of the system.\u201d\n\nTo protect against misleading insights, learn to think critically and don\u2019t immediately leap to what seems to be the most obvious conclusion. As business processes and data architectures grow larger and more complex, many things can go wrong, and finding the root cause can be tricky. \u201cThe best approach is to surround yourself with a diverse team of subject matter experts to consult with before making decisions,\u201d Gelvan advises.