Hindsight may be 20\/20, but so is knowing when to cut your losses before investing too much money, time and effort on a project that won\u2019t achieve the desired ROI.\n\nThat\u2019s what Toronto-based real estate services and investment management firm Collier International has learned. Rather than committing to new projects based on traditional requirements gathering and early estimations, the global IT group starts with short sprints designed to validate a business idea, the technical solution and the potential business benefits, says Mihai Strusievici, vice president of global IT.\n\n\u201cThe incremental sprints approach provides cost certainty and a go\/no-go decision point between sprints,\u2019\u2019 he says. \u201cThis way, the development team learns as it goes at a known burn rate and has the opportunity to stop before too much effort is spent on something potentially not useful.\u201d\n\nWith the pressure on organizations to roll out digital initiatives quickly, it\u2019s not surprising that sometimes progress gets stalled and outcomes don\u2019t meet expectations. Even when IT leaders have estimated and measured what ROI should come from a project, often they have to pivot and make changes to maximize their investment.\n\nThe average digital transformation stands a 45 percent chance of delivering less profit than expected, while the likelihood of surpassing profit expectations, on average, is just one in 10, according to McKinsey.\n\nThis is often the case simply because you don\u2019t know what you don\u2019t know, industry leaders say. The key is to set tangible goals and establish practices that ensure clarity and communication every step along the way. Whether your organization seeks to grow by 30 percent, expand its market to another continent, or increase its number of customers, you need to have \u2014 and communicate \u2014 key indicators for each goal, says M. Nadia Vincent, a digital transformation and AI executive advisor.\n\n\u201cThen you have to have milestones by which you want to achieve those metrics,\u2019\u2019 she adds. \u201cThen monitor what\u2019s happening with any project management tool. It\u2019s the strategy that matters \u2014 not the tool.\u201d\n\nOnly by measuring progress can you determine whether you are meeting your target, need to raise your level of activity, or change your strategy altogether, Vincent says.\n\n\u201cSeeing changes in revenue or margins is one litmus test for success, but companies should also regularly measure the customer and employee experience to ensure the technologies are having the right impact,\u2019\u2019 agrees Tom Puthiyamadam, global digital and BXT leader for PwC.\n\nFollowing are lessons organizations gleaned as they measured the progress of their digital initiatives. The common denominator? Speed and an agile approach to development.\n\nLearning to pivot\n\nMcKinsey research shows that 66 percent of software projects go over the budget, largely due to unclear objectives and shifting requirements, Colliers\u2019 Strusievici notes.\n\n\u201cIn other words, in the context of moving targets, the initial estimate is more of a guess than science, a fact described by the Cone of Uncertainty Theory developed by Barry Boehm in the early 1980s,\u2019\u2019 he says. After all, if your end product is not well defined, \u201can initial estimate is impossible.\u201d \n\nTo avoid this pitfall, Colliers global IT takes a measured approach that involves clarifying the goal as the project evolves, along with the opportunity to stop before officials commit too much money and time, Strusievici says.\n\n\u201cWe start by investing in a functional prototype that allows us to get feedback from real users,\u201d he says. \u201cAs we learn, we either pivot \u2026 or stop, saving the effort that would have been wasted in something potentially useless.\u201d\n\nChanging direction, or pivoting, is a regular occurrence in all projects, and is a result of learning and adapting to new information, he says. But in the traditional approach, pivot equals \u201cchange request,\u201d which usually means increased cost, Strusievici says.\n\n\u201cIn an agile, incremental process, the pivot is expected, even celebrated, and the team is ready to make decisions without the pressure of staying on a predetermined course,\u201d he says. \n\nThe waterfall methodology has its merits when you know what you\u2019re going to do, he adds, because it is predictable and reduces ambiguity. But a lot of digital transformations are experiments, and organizations don\u2019t know if they will resonate with users.\n\n\u201cWhat we learned over the years is that we need to figure out very early the context on which we operate,\u201d Strusievici says. \u201cIf it is experimental, unclear [or] aspirational, we need to adopt an incremental, agile approach and be ready to pivot as we learn. Not every digital initiative has to [be completed] and stopping early is as valuable as completing on time.\u201d\n\nThe real magic of measurement\n\nWhen Toyota Financial Services decided to get into the private car financing business, IT was tasked with developing a new technology stack comprised of more than 15 systems in less than six months to service Mazda, its first client. Officials opted for a \u201cspeed of decision-making\u201d approach using an agile-inspired methodology, says Amit Shroff, vice president, office of the CIO, at Toyota Financial Services.\n\n\u201cWith any system there\u2019s a process, the business objective and how do I digitize that process, which requires people from a multidisciplinary effort,\u2019\u2019 he says. \u201cOur idea was really to focus on breaking that up into areas of expertise and focusing more on what decisions need to be made.\u201d\n\nThe business and IT \u201csquads\u201d met daily or multiple times a week to deal with any project impediments. Senior management met with the squads at least weekly to inject a sense of urgency around transparency on the issues, Shroff says.\n\n\u201cWhat we realized is there is really a need for \u2026 a level of transparency to information. Nothing else matters.\u201d That way, decisions get made on how to proceed more quickly, he says.\n\nThere also has to be a \u201chyper focus\u201d on what is a hurdle that can\u2019t be solved by the team so it can be elevated to the next level. \u201cThat\u2019s huge. Very few companies do that,\u2019\u2019 he says. \u201cOnce you have that transparency \u2014 and it requires some level of confidence and risk taking \u2014 then you have people who can \u2026 reallocate resources or substitute if someone doesn\u2019t have right expertise.\u201d\n\nPeople tend to have a microfocus on achieving ROI, Shroff says. But the key is to have \u201ca confidence level on the return and purpose of the return. That\u2019s the magic. What is the purpose and why are you doing the digital transformation. If it\u2019s just purely financial, why do it?\u201d\n\nMost projects focus on time and money, he adds. \u201cWe\u2019re saying if you focus on moving these things forward, the time and money will be a good outcome,\u2019\u2019 Shroff says. \u201cWe can manage time itself, and the money you can manage by moving quicker.\u201d\n\nMeasuring customer feedback and social media ratings\n\nCustomer-facing businesses care deeply about user ratings. NJ Transit deployed an app a couple of years ago that was \u201cnot customer friendly or digital friendly,\u2019\u2019 says Lookman Fazal, chief information and digital officer for the transit system, the nation\u2019s third largest agency, which transports one million people a day.\n\nIT added what Fazal says was missing functionality: integration with Apple Pay so users could more seamlessly purchase tickets, and a simpler user interface that wasn\u2019t \u201csmall and ugly,\u201d he says. Several other features were also added, including notifying riders about the location of trains and buses and letting them save their favorite routes.\n\nInstead of pushing the app out to its 800,000 customers right away, Fazal says they deployed it early to internal testers, then gave a \u201csneak peek\u201d to the digital community for testing and feedback purposes.\n\nIT used the agile methodology and conducted short sprints, he says. \u201cNow, we could address people concerns and issues in a week to two weeks rather than months.\u201d\n\nThe resulting feedback was a \u201cwhole pipeline of requests\u201d once IT has rolled out the native apps for Android and iPhone as opposed to having a single app, Fazal says.\n\nIT measured the apps\u2019 ratings along with feedback from riders and how fast people were interested in downloading either version. Roughly 70 percent have so far downloaded the iPhone app while 30 percent have downloaded the Android one, he says.\n\nThe ROI ultimately met what IT measured, Fazal says.\n\n\u201cThe ROI was not from a dollar perspective, but we did recover development costs \u2026 it was more on the ratings,\u2019\u2019 he explains. \u201cWe didn\u2019t want to roll out any app that would give us less than a 4 rating [out of 5].\u201d\n\nThe proof was in the pudding. \u201cWe exceeded it: It\u2019s at 4.7\u201d in the Apple store, compared to the previous 2.3 rating for the old app, he says.\n\nMeasuring manual labor and user satisfaction\n\nWhen you\u2019re in the business of helping children, the last thing you want is for employees to have to spend time away from their core mission. New York Foundling, one of New York City's oldest and largest child welfare agencies, found that employees were spending too much time on repetitive, data entry because state and local systems were not integrated. That led to a 42 percent turnover rate in 2017, says Arik Hill, CIO of New York Foundling.\n\nThe agency collects about five million attributes on clients each year and staff were spending too much time inputting data into five disparate systems, which also created inconsistencies, and led to significant delays in communication, Hill says.\n\nA survey revealed that social services staff were spending up to four hours a week copying and pasting data into these systems. \u201cThis directly impacted the job retention and satisfaction rate,\u2019\u2019 Hill says.\n\nThe decision was made to invest in a robotic process automation (RPA) system, utilizing an agile approach, forming teams comprised of clinical users, supervisors and technical staff, he says.\n\nThe teams went through a series of agile development cycles and came up with \u201coversight activities\u201d to ensure the quality of data going through the systems was being automated correctly.\n\nIT knew staff were spending four hours a week doing data entry so the goal was to eliminate that completely, Hill says. After an action was completed by a bot during a supervised robot development cycle, the team received a response back.\n\nOne of the issues that cropped up during development was lag time in the city\u2019s system for inputting a new client, which impeded the bot\u2019s ability to record transactions. That meant staff still had to manually input information.\n\nIT monitored how long it took the bot to complete a transaction, as well as how many transactions the bot couldn\u2019t complete. Hill says officials have been tuning the timeframes so that 100 percent of the transactions flow through without human intervention. The goal is for the system to run in near real-time.\n\nStaff productivity was also measured. Today, staff turnover has decreased to 18 percent, Hill says.\n\nWith the RPA system, \u201cyou have to have a good grasp of your baseline metrics in knowing where to target the tool,\u2019\u2019 he says. \u201cYou can do a lot of things, but there are certain areas that are overly complex that don\u2019t lend themselves to automation. You want things with a predictable outcome that are constantly happening.\u201d\n\nMeasuring agent backlogs, customer satisfaction\n\nThe focal point of digital transformation at Liberty Mutual Insurance, the country\u2019s fifth largest personal insurance carrier, has been moving from traditional IT metrics and a waterfall approach to more business-focused metrics, says Andrew Palmer, CIO of U.S. retail markets.\n\n\u201cWe used to size 1,000 projects and hope we got business results,\u201d he says. \u201cNow, it\u2019s a more iterative process\u201d that involves teams comprised of business users and IT.\n\nOne such digital project involved creating a capability for customers to text photos to agents to start the claims process instead of them calling into the call center.\n\nThe agile approach included incremental development that considered feedback from agents on what would make their lives easier, Palmer says. \u201cAlong the way, we made dozens of pivots in terms of what we thought would be effective versus what resonated\u201d with the agents, rather than hearing from business analysts speaking on their behalf, he says.\n\n\u201cIt helped us get much closer to customers and prioritize backlog,\u2019\u2019 he says. By doing \u201cdozens of releases a day\u201d now, the software being built is simpler and more usable, he says.\n\n\u201cIf we had done that project two years ago we might have done it over seven months as a big-bang release among agents and it may not have met the need,\u2019\u2019 Palmer says.\n\nIt\u2019s one thing to have a hypothesis that a customer would want to take a picture of a car after an accident and text it to their agent, he notes. Like the others, Palmer says the way to deal with an unknown is to conduct a pilot test and pick some metrics to measure before building out all the infrastructure.\n\n\u201cIn the past, we\u2019d say we\u2019d have to ingest all these images and spend time and money on putting that project into the market without knowing,\u201d he says.\n\nNow, with a modern architecture, Palmer says instead of doing maybe 12 software releases a year, IT does several a month using \u201cmicro tests\u201d to hone in on the customer experience. All customer feedback is fed into a Slack channel so project teams can react accordingly, he says.\n\n\u201cHow we accept risk and have teams win or lose together is part of building that strong business-IT relationship,\u2019\u2019 Palmer says. If a team works on something for a week and finds it has to regroup, \u201cit\u2019s not a big deal.\u201d That is also the case if something doesn\u2019t perform when it is put in production, he adds. \n\n\u201cWe\u2019re much more comfortable stopping projects if something\u2019s not working big and small \u2014 it\u2019s no longer the failure of the team, it\u2019s just that we need to reallocate resources in a different direction,\u2019\u2019 he says.\n\nWhereas people used to get attached to projects and outcomes done over a long period, now it\u2019s easier to redirect teams to different business objectives, he says.\n\n\u201cIt\u2019s about trying to align teams to business outcomes and creating a risk-tolerant environment,\u201d and adopting the agile principles while adapting the practices. Palmer says the biggest switch has been moving from a \u201cone-size-fits-all mindset\u201d to one of a \u201ctest and learn and pivot mindset.\u201d\n\nLiberty Mutual\u2019s U.S. Personal Insurance Technology team has seen employee engagement rise over 50 points as measured by Net Promoter Score, he says. Delivery productivity of new capabilities has improved by 30 percent, Palmer adds.