6 tips for working with AI vendors

For most organizations, DIY artificial intelligence is out of reach. Here’s how to cut through the hype and create business value with off-the-shelf AI.

6 tips for working with AI vendors
xijian / Getty Images

Artificial intelligence and related technologies promise to be a game changer for businesses. The central question is whether to build or buy. Some companies will build their own solutions from scratch, but using commercial tools can often be faster, cheaper, and provide better results than what a company can build on its own.

Since the technology is so new and untested, early adopters have some power when it comes to vendor relationships. Suppliers may be willing to perform custom configurations and integrations, provide free consulting or training, or give steep discounts — especially if you’re willing to be one of their success stories should the project work.

Plus, being among the first customers of a startup, or for a new offering from an established vendor, may allow you to impact how a product develops.

But working with AI vendors isn’t all wine and roses. In addition to the usual issues that may arise when working with any technology vendor, there are specific nuances when it comes to AI, machine learning, and similar technologies.

Beware the hype

Alight Solutions, an HR services company based in Lincolnshire, Ill., began looking for an AI-powered chatbot in 2014.

The company handles a wide range of HR benefits for 3,000 enterprise customers, which combined employ more than 23 million people. That’s a lot of calls and web chats with support representatives. Many of the questions are repetitive, but the answers can be different for each employee.

An AI system would not only have to be able to understand the question but find the specific answer for that particular person — and be smart enough to escalate to a human representative if it failed.

But when Alight began talking to vendors, the company saw a lot of hype in the market. Vendors were promising natural language processing systems that were ready to go out of the box.

“When you go and test their system, you'll find that even with small changes in wording the system would get tripped up,” says Ibrahim Khoury, the company’s director of product development and innovations.

This hype issue hasn’t gotten any better, he says. “I think there’s more fluff today, to be honest, with the marketing of AI and machine learning.”

Be prepared for a serious learning curve

AI systems require training, says Khoury. They don’t work right out of the box. But some vendors promise systems that work with no human intervention at all, he says. Alight wound up going with a company that didn’t make those kinds of promises.

“They were up front with us,” he says. “They said, ‘Let’s not kid ourselves. You are going to need to put people and processes in place to make sure that the system works. You’ll have to train it, monitor it, invest money in making sure that it works.’ That honestly woke us up, because no other vendor was telling us that.”

In fact, after Alight decided to go with its vendor, Verint, in the fall of 2016, it took about half a year before the system was ready to go live.

“There are a lot of variations between one client to another,” he says. “And even within one client, you have different levels of employees. You might have a management group, a union group, an hourly group. We had to build a system that can vary the answer based on the individual.”

That required a team of subject matter experts in the services that Alight offered.

“We’re not an industry that goes by answering things wrong,” Khoury says. “We want to make sure that, no matter what the answer is, it has to be accurate. And if we can’t answer it, we have to build an escalation model, so we don’t leave you hanging. If the service center is open, let me hand you over to a web chat agent. If not, let me schedule an appointment so that an agent can call you back.”

It took time to get that level of knowledge and integration with the call center operations in place.

Pilot, pilot, pilot

But before Alight moved ahead with the project, it first conducted an extensive pilot with Verint.

“We tested the systems of other vendors, but the effort to do the pilot itself was significant,” Khoury says. “It required a lot of security, architectural updates and changes. It required a lot of implementation on our side. So, after we tested several systems, we focused on one pilot. That gave us the results that convinced us that this was the path forward.”

Putting the vendor to the test was also key for Priam Capital, which buys and manages multi-tenant office buildings in the Midwest and Southeast.

The challenge the company had was that its employees were spending too much time dealing with the paperwork, and not enough time buying buildings. Commercial leases are extremely complicated, sometimes over a hundred pages long, with hundreds of individually-negotiated details. On top of that, all older leases are on paper.

Many vendors offer AI-powered optical character recognition and natural language processing, says Abhishek Mathur, the company’s founder and CEO. Often, the technology doesn’t work, or doesn’t work for this particular use case. “There’s a lot of smoke and mirrors out there,” he says.

So, two years ago, when Priam was selecting a vendor, it pulled together its most difficult leases to see whether vendors could handle them. The vendor the company chose, Leverton, had the specific real estate industry expertise required to identify the key elements of the leases, he says. “There was really no one else who was this focused on real estate leases.”

That became clear during the testing.

“When we went out and tried a lot of different systems, they were not able to pick up some of the nuances that were very significant from a revenue perspective,” he says. “I recommend that people try it before they buy it.”

Consider transparency

No AI system is perfect. But, with AI systems, it can be hard to tell whether the output is misleading because the internal algorithms may be completely inscrutable.

Because a mistake with a commercial lease could result in significant financial losses, this “black box” problem was a concern for Priam, as it is for many companies. To address this, Leverton’s tools pull in all the details from leases but also include a link back to the text in the original document.

“There’s no way that we could have an AI do everything right,” says Leverton CEO Abhinav Somani. “There are certain nuances or strange terms that are challenging for anyone to pick up.”

There’s a partnership between the system and the asset managers who use it, he says. In addition to allowing asset managers to quickly review details in the contracts, the system will also highlight areas that need manual attention.

There’s a lot of research going on right now in explainability, transparency, and also in understanding the built-in biases of AI systems, says Lauren Neal, principal at Booz Allen Hamilton. The more information companies have about how their vendors’ models were built and trained, she says, the more they can understand what that models’ limitations are.

Jennifer Fernick, head of research at NCC Group, a cybersecurity advisory firm, warns against working with vendors that are too secretive about their AI models.

“In general, any vendor who is unwilling to talk about their algorithms and implementations is probably not doing anything interesting,” she says. That doesn’t mean that customers can demand to see the source code, she says, but vendors should be willing to talk about the types of algorithmic tools they’re using, whether they’re based on open source libraries, and how they will be applied to data.

“There is an inverse correlation between the claim that they ‘can’t show you the secret sauce’ and the chance that there is anything particularly innovative in their product,” she says.

Another aspect of transparency, one that should be top of mind of any company today, is that of privacy and security. AI systems, in particular, are extremely data-hungry, and if that data is sensitive, it can put a company at risk.

“You are only as secure as your weakest vendor,” Fernick says.

For example, does the vendor require you to move data into their cloud, and doesn’t have a good security team in place? Is the data visible in plain text to the vendor’s employees?

“There are many considerations beyond the analytical capabilities of the tools themselves,” she says.

For highly sensitive data, companies might consider a vendor that offers an on-premises solution.

Focus on business value

Waqqas Mahmood, director of advanced technology and innovation at Baker Tilly Virchow Krause, came to the accounting and consulting firm a year and a half ago to lead its digital transformation strategy.

Baker Tilly, which employs about 4,000 people in its global workforce, had undergone multiple mergers, resulting in a lot of processes that needed to be standardized and made more efficient. In addition, Baker Tilly wanted to implement a chatbot system for tech support tickets.

The company couldn’t do it on its own, says Mahmood. “There was a lack of skills,” he says. “And the sheer time it would have taken to build it internally — it would have taken a long time.”

For Mahmood, the key to making an AI vendor relationship work was picking projects with concrete goals. “If you don't have the right business case, and the right need before implementing AI, it's not going to be successful,” he says.

For workflow automation, Baker Tilly decided on NICE Robotic Automation, a relationship that is already paying off, he says, with automated tasks taking a third as much time to complete as they did before.

“We have decreased our chargeable hours quite a bit,” he says. “In the double digits, percentage-wise. We’re trying to do the same job with higher quality in less time — we’re not looking for the short-term gain of charging for more hours. We want to provide a high-value service for our clients.”

The long-term goal of its workflow automation strategy is to add predictive analysis — not just automating processes and flagging items, but helping with decision-making, which would eliminate some human work.

“That will have an effect on FTE,” he says. “So we can put people into more high-value work.”

As for tech support, Baker Tilly’s deployment of a chatbot from Bold360 took about twelve weeks and has reduced tech support hours by 35 percent, Mahmood says. “We spent about 80,000 hours a year on ticket resolution, people calling in to get their password reset, get a new printer, or something isn’t working and they need to download new software,” he says. The chatbot understands the request and routes it appropriately, says Mahmood. “It’s not sexy, but it takes care of our back office pain.”

Both projects are part of a long-term plan, he adds. “We are creating a holistic data strategy for the firm, how we manage data, how we use it, what the architecture should look like,” he says. “We are creating a platform that will have an analytics engine in the middle with AI capabilities.”

Be patient

Mahmood has mapped out Baker Tilly’s AI journey, and is currently still in the early phases. “It takes a lot of patience,” he says.

Everyone always underestimates how much effort it will take, says principal analyst and founder at Constellation Research, and how much money.

“The biggest problem that customers typically face is they want to do these projects, but they're only funded to do proof of concept,” he says. “But these are long-term projects.”

In addition, he says, if a company picks a starting project that doesn’t have a good business case behind it, people will lose interest.

“It happens to a lot of companies, and you waste a lot of money,” he says.

Copyright © 2019 IDG Communications, Inc.

Survey says! Share your insights in our 19th annual State of the CIO study