by Clint Boulton

7 tips for battle-testing new technologies

May 16, 2018
Emerging TechnologyEnterprise ApplicationsIT Leadership

CIOs offer recommendations for conducting proof-of-concept technology tests, as well as vendor u201cbake-offs.u201d

face off boxing battle man in the middle
Credit: Thinkstock

Courting new technologies is a risk for IT leaders, who must wisely allocate finite budgets to hardware, software and services they believe will support business operations. Bad bets can set the company back years and ultimately prove catastrophic for CIOs.

This is particularly true in the digital era, as a proliferation of newer technologies such as cloud, analytics and machine learning render vendor selection more vital than ever. To select the best tools for the tasks, CIOs conduct “proof-of-concepts” (PoCs), essentially pilot tests in which they model how technologies will run in their organizations.

Frequently these experiments include “bake-offs,” or competitions between vendors with similar solutions. Technologies that don’t run well are quickly jettisoned in an era where failing fast is fashionable and getting a viable solution into production is the ultimate goal.

CIOs recently shared with details about PoCs and bakeoffs they have conducted, along with tips on how other IT leaders should approach battle-testing new tech.

Incumbents vs. upstarts

Shortly after joining AutoNation to run IT infrastructure in 2016, Adam Rasner realized he had a serious data management problem. The car and automotive company suffered thousands of failed backup jobs per month as AutoNation’s data growth taxed legacy storage systems.

Rasner initiated a bake-off between one of AutoNations’ incumbent storage vendors, as well as two emerging, yet unproven startups. It was a dicey situation with no clear resolution. Rasner knew the incumbent might not scale to meet AutoNations’ data demands, while analysts warned him that endorsing either of the two startups, neither of which had many Fortune 100 references at the time, was risky. Even so, Rasner says that he tested the three solutions side by side on the same equipment in an AutoNation data center.

He wrote the incumbent off when it hit a limit that would incur additional hardware investment. One of the startups shot itself in the foot when it falsely reported a backup job as complete but lost the data in the process. Although the vendor patched the problem with firmware, the damage was done. “You have an issue like that it’s really hard to get that confidence back,” Rasner says. “They couldn’t recover from that.”

Cohesity emerged as the winner, ultimately reducing the per-terabyte storage cost for AutoNation by 40 percent. Rasner says he’s since extended use of Cohesity technology to secondary storage for document management, and plans to boost his investment in the technology as he scales the company’s infrastructure. Cohesity, Rasner says, has met AutoNation’s demands every step of the way.

Tip: Vet your vendors

Before inviting new companies for a PoC or bake-off, inquire about their current customers and their financial security, Rasner says. For example, AutoNation sent its procurement team to check the financials of all three vendors financials and report back to IT.

“We’ve taken a few risks on a few different products, and it’s turned out pretty well — you just have to careful,” Rasner says. “You want to be innovative and try new things, but you also don’t want to put the company’s money on the line when it’s not rock-solid.”

Betting on unproven tech

Zoom CIO Harry Moseley has initiated more PoCs than he cares to count. Not just at Zoom, but also in his previous capacity as CIO of organizations such as Blackstone Group and KPMG. But one proof-of-concept stands out for Moseley: an early test of Amazon Web Services several years before many organizations were even thinking of thinking of endorsing the cloud. His team performed penetration testing and code reviews prior to uploading thousands of documents into AWS. While that bet paid off, Moseley says he’s also tested commercial software only to reverse course and build proprietary applications for a fraction of the cost.

In short, the PoC is a test bed with no guarantee of success, intended to let a company discover the art of the possible. Moseley advises CIOs to:

Tip: Assess your risk appetite

CIOs working for companies in regulated industries can’t afford to bet the company on unproven technologies for critical business functions. For example, an enterprise working in a heavily regulated field won’t able to stomach too much risk. “Some organizations don’t want to be on the bleeding edge,” Moseley says.

Tip: Influence product roadmaps

But it sometimes pays to give startups a chance, as it affords CIOs the opportunity to get better pricing and potentially influence a product roadmap for years to come. “You have a lot of leverage to steer and guide the platform,” Moseley says. “It’s akin to having a seat at the table.”

Sniffing out snake oil

National Life CIO Tom Anfuso says the insurance company does a lot of its own homework for its PoCs, which recently included a data service platform, as well as new policy administration and access control solutions. Anfuso recommends:

Tip: Beware lightweight MVPs

Anfuso says CIOs should direct the structure of vendor mock-ups in PoCs. For example, some vendors may show little functionality in a minimum viable product, which should immediately raise suspicions; the vendor could be steering CIOs toward a low hurdle simply to get them to buy the product.

In general, CIOs should take care not to use the MVP concept to guide large-scale systems integrations projects. “The graveyard of enterprise IT is littered with the corpses of MVPs that were supposed to become much bigger,” Anfuso says.  

Tip: Practice good faith in bake-offs

Sometimes enterprises are the ones pulling the rope-a-dope, pitting multiple companies against each other to force a better deal. But Anfuso says bake-offs must be done in good faith, and CIOs shouldn’t put any company through its paces unless they are truly interested in them. “You can get a bad name, when you have zero interest selecting [a vendor],” Anfuso says.

The importance of metrics

TiVo CIO Steve Palmucci finds himself regularly running PoCs and bake-offs as he re-platforms IT around best-of-breed tools in the wake of Rovi’s acquisition of TiVo two years ago. Palmucci has conducted PoCs for anything from collaboration and communication tools to security, server and storage systems.

Palmucci says CIOs should:

Tip: Loop in the business

Many PoCs are completed within the context of IT, but Palmucci likes to extend PoCs to more functional areas of the business. This gives him a better idea of how a tool will impact a business user. For example, TiVo executives recently gave new video conferencing technology a spin for a staff meeting.

Tip: Measuring is a must

While it’s not necessary to have a detailed playbook or checklist to conduct a PoC, IT should introduce structured ways to gauge test results, including quantitative and qualitative measurements. Quantitative testing will help CIOs determine functionality, performance and fit within the business. Qualitative testing takes into account skillsets within IT to administer and manage a technology, as well as how willing a vendor is to provide support if needed.