by Clint Boulton

Workforce analytics: Measuring employee productivity in the agile era

Jul 02, 2019
AnalyticsBudgetingIT Leadership

CIOs seeking to quantify employee performance and productivity are turning to analytics with business value benchmarks baked in — but can collaboration and agility be accurately gauged?

gears automation robotics ai machine learning woman with binary code
Credit: bernie_photo / Getty Images

As IT departments walk the tightrope between boosting business outcomes and managing costs, it’s become more crucial than ever for CIOs to gauge the productivity of their staff. How did Bob and Jill perform in the value stream while delivering product X to the business?

There’s just one catch: CIOs have long lacked efficient mechanisms to do this. In a digital era drenched with tech that enables us to hail rides, control lights and file expense reports from a smartphone, most CIOs can’t accurately measure the productivity and quality of the contributions of their staffs.

While many IT leaders are addressing this data gap with do-it-yourself (DIY) approaches, or relying on old-school self-evaluations and performance reviews, startups are producing so-called “workforce analytics” tools to help mitigate the issue.

Equating employee workflows to server workloads

Think of it this way: IT leaders can calculate the cost to run each server or storage array against the value it delivers. But some IT leaders are hungry for a way to analyze employee performance, as if they were a server to be tuned, reconfigured or redeployed based on business requirements.

Benchmarking job performance versus the cost of each employee is something with which Tyson Foods CTO Scott Spradley has been wrestling because each role at the chicken production giant is different. For instance, the value ratio varies between a data scientist and a software engineer. “It’s about taking a taxonomical value approach to each person in the workforce chain,” says Spradley.

To date, approaches to quantifying labor have been old school. Companies pay McKinsey, Bain & Co. and other big-name consultants millions of dollars to assess workforce performance. They assign professional consultants to shadow employees for weeks or months, take notes, write up evaluations, make recommendations and move on to the next gig.

But the value of this approach has come into question as digital technologies have disrupted every business sector. It’s no longer enough to simply clear Net Promoter Score targets or look inward to crunch employee satisfaction metrics.

Simply, the nature of work has changed. A decade ago companies built software in “waterfall” cycles, taking 18 to 24 months to ship products that were lucky to meet the businesses’ needs. Today companies deploy “agile” teams that build software in one- to two-week sprints. IT and business siloes have yielded to cross-functional teams, with software programmers and UX designers co-located with product managers and business analysts. This confluence of speed and personnel hodgepodges can confound the savviest of consultants.

CIOs need new tools that measures output down at the operational process and, yes, even task level. “We need a different style of management than how much work we can crank out,” says Gartner analyst Bill Swanton. The phrase “workforce analytics” will do.

DIY analytics approaches

Many CIOs are embracing the do-it-yourself approach to workforce analytics.

While working as CIO of Aflac, Julia Davis created baseline metrics for measuring productivity, such as tasks done per full-time employee, per month. Tasks ranged from help desk requests, such as fixing printers, to code changes. When Davis progressed to analyzing application development, she ran into challenges measuring the individual performer in agile. However, consistent with bottoms-up approaches to management, Aflac’s “self-policing” agile teams kept schedules and team members on track, says Davis, who retired from the company in 2018.

For the past two years, TD Ameritrade has used an internal methodology and tool it calls Go Live, which measures the business functionality generated by an agile team on a given sprint, says CIO Vijay Sankaran. Initially, Go Live measured the number of features generated for each app, but has since expanded to gauge the performance and productivity of each engineer to assess whether they are improving over time.

Encouraged by the results, Sankaran’s team is now broadening Go Live to gauge the various wait states for an application as it wends its way through an agile development cycle. Go Live scrutinizes an app from conception to story creation to code completion and so forth, all the way to its launch into production. Finding room for improvement, the team has since created an Objective and Key Results (OKR) metric to optimize the idea-to-production pipeline.

“There are so many different operational levers to think through; what do you use, where do you place optimizable resources on agile teams, and at what investment levels,” Sankaran says. “This art needs to be demystified, but it requires a big cultural change on the line-of-business side.”

To analyze how TD Ameritrade uses software and hardware in total, Sankaran is also adopting technology business management (TBM), an increasingly popular methodology of aligning the cost to manage IT with business value.

Vendors set sites on workforce analytics

IT departments have historically used spreadsheets to execute these analyses, but many are jettisoning this approach in favor of TBM analytics software from Apptio, which has also begun to tackle the thorny problem of scrutinizing labor costs.

Recognizing that labor costs account for 50 percent of the cost structure of technology in an enterprise, Apptio last year launched Agile Insights, a SaaS solution that measures value delivery, cost of quality and labor utilization.

Companies can use the tool to, for instance, analyze code check-ins, cost-per-storypoints and other metrics generated through agile sprints in project management apps such as Atlassian’s Jira and benchmark it against the financials in their ERP systems, Apptio CEO Sunny Gupta says. “It’s about getting out of the anecdotal and getting into informed, quantitative decision-making,” he says.

Other startups are joining the fray. Fin Analytics is building software that seeks to help enterprise better understand how employees complete their daily tasks. The company, venture-backed by Kleiner Perkins, Accel, and CRV, is selling a Chrome browser plug-in that records employees’ mouse scrolls, website clicks and browsing behavior, Fin co-founder Sam Lessin tells It also tracks application consumption, such as how much time each employee spends in Slack, or how much time they spent poring over a slide deck presentation. The tool records video and audio, creating a moment-by-moment process catalog.

Fin’s software serves this information up in statistical dashboards, highlighting key metrics that managers can use to coach team members.

Lessin says that taking the process-level approach to improving productivity and performance is a differentiator from most analytics, which measure business outcome metrics, such as NPS and CSAT (customer satisfaction) scores. “No one has gone super deep on the process measurement to understand how the things we’re doing are impacting our work,” Lessin says, adding that dozens of customers use Fin today.

Healthy skepticism around workforce analytics

Such a quantified approach to the workforce poses its challenges. Anna Frazzetto, chief digital technology officer of Harvey Nash, recalls efforts companies took a decade ago to determine the efficacy of software programmers. One programmer may program a widget writing 1,000 lines of code, while another may program the same tool writing 200 lines. But if the widget crafted in 200 lines has more defects, the efficiency factor goes out the window. “In my humble opinion, they haven’t been able to solve the problem,” Frazzetto says.

It’s also hard to compare teams of people working on different products, says Swanton, the Gartner analyst. Given varying degrees of difficulty, a UX developer working on one product can’t be evaluated the same as one working on another product. Ditto someone developing an AI algorithm, whom a CIO would hardly ask, “How many lines of code did you write today?”

Moreover, Swanton says CIOs who try to measure worker performance run the risk of seeming as if they are rewarding or punishing people in a way that discourages teamwork, a veritable death sentence for agile.

Swanton’s recommendations for measuring output harkens back to something most CIOs are laser-focused on today: Judge each product or agile team on the value that is created, or the business outcome achieved. Does the product deliver a measurable improvement in business capability? Does it generate revenue or curb costs? If it does any of these things, it’s a win, Swanton says. “The goal of IT is to make the business work better,” Swanton says. “That’s the metric that really matters.”

While Lessin acknowledges some corporate leaders may be concerned about scaring employees with this virtual over-the-shoulder approach to process improvement, he says the heightened emphasis on providing superior customer service will win over skeptics and appeal to data-driven CIOs.