Making the most of sentiment analysis

Sentiment analysis is beginning to prove its mettle in the enterprise. This analytic technique, which enables companies to determine the emotional value of communications, is finding traction in a range of use cases, from meeting transcription to customer service and feedback.

These days, sentiment analysis relies largely on supervised or semi-supervised machine learning algorithms. All the big cloud players offer sentiment analysis tools, as do most major customer support platforms and marketing vendors. Conversational AI vendors also include sentiment analysis features in their wares.

But making the most of sentiment analysis requires a curious blend of art and science. Here is a look at how some organizations are putting sentiment analysis to beneficial use.

Underscoring importance in transcription

Most virtual meeting platforms offer transcription services. In fact, voice recognition is built into a lot of what Microsoft and Google offer out of the box. Zoom also plans to offer live transcription this fall but, until then, there are third-party services such as Otter AI.

But computer transcription is a poor substitute for a human note-taker because human judgment is necessary to ascertain importance versus idle chit-chat, and to figure out what the next steps are, and who committed to what.

To address this gap, transcription provider Pickle is turning to sentiment analysis. The Pickle platform uses AssemblyAI, a speech-to-text API, for its transcription functionality. But while some open-source tools can perform sentiment analysis, they tend to focus on identifying particular keywords, says Pickle CEO and founder Birch Eve. Because of this, the company decided to build its sentiment analysis machine learning models from scratch.

Pickle’s approach uses a supervised learning model in combination with unsupervised learning classification algorithms. For the supervised portion, humans — originally Pickle employees — would manually label and classify parts of conversations, including differentiating between casual chit-chat and important business. They also noted any conversational segments that had a strong positive or negative emotion. As the company ramped up, it turned to Scale AI to do more of the labeling and classification.

The training data set grew to a million conversations, Eve says, and the first generation of models had accuracy levels between 77% and 83%, he says, depending on the type of conversation being analyzed.

“We do quality control where we randomly flag conversations and send them to a queue where we go through them manually and double-check the model,” he says. “If something is off, we go back to the model, see where the inconsistencies are, and either tune the data or switch out the data sets.”

Today, accuracy is at 93% to 94%, he says. There’s less variation in part because since January the company has been focusing on Zoom conversations. “It’s made the data more consistent because most Zoom conversations have a similar style,” he says. “There’s some small talk, and then the business side.”

The key to success with AI projects involving sentiment analysis is to stay focused, Eve says.

“It’s exciting when you start getting consistent data back and start looking into other areas you can do, and there’s a couple of trap doors we fell into,” he says. “But the best path to success is to keep your head down and focus on sentiment alone.”

Capitalizing on product reviews

The practice of sentiment analysis goes back 15 years, says John Dubois, principal in Ernst & Young Technology Consulting. Back then, it followed the “bag of words” approach, which simply counted up how many times particular words appeared in a conversation, social media post, news article, or product review.

“The outcome was thumbs up or thumbs down,” he says. “It’s since changed quite a bit.”

But machine learning is helping organizations better determine the sentiment behind those words. One very fertile area where sentiment analysis machine learning models is having an impact is in product reviews, as a review could be extremely positive, or negative, without using words like “great” or “terrible” — or it could use those words in a sarcastic way.

For example, consider this review: “I really thought this dress was going to be fantastic. The pictures were beautiful, and the packaging it arrived in was just perfect. Then, when I put it on, I looked like a jolly giraffe. But at least my dog likes to sleep on it, and seeing my dog happy makes me happy.”

There are a lot of positive words in this review, but the attributed star rating is low. Ratings are, in effect, a sentiment score — what the customer thinks of the product overall. As machine learning systems read and compare millions of reviews against the ratings buyers give, they get better at understanding the real emotions behind the words.

Dubois recently worked on a project for a domestic car company that involved analyzing sentiment related to car features for all the major automotive manufacturers. For example, customers might like the cup holders in one model but dislike the competitors’ version, or like the competitors’ interiors better than those of the client.

Analyzing sentiment enabled the company to determine exactly what customers liked about its products, and where it lagged behind its competitors — analysis that became fuel for better advertising. “As we’re going to various car buying and evaluation spaces, we can buy ads based on where we know our strengths are and their weaknesses are,” Dubois says.

“We did that for spring, summer, and fall sales events and we saw a 15% increase in click-through rates and a 4 to 6% increase in conversion,” he says. “And 4% in auto is a pretty big deal for a sales event.”

There’s quite a bit of AI being used for sentiment analysis for these kinds of point solutions, he says. But companies may find even more value in sentiment analysis as a strategic weapon if they go beyond the department level.

“It could be used by sales to update product listings,” he says. “It could be used by merchandising to update the information architecture of a site. It could be used by R&D.”

Once the technology has proved itself, IT leaders should take it to a broader group of business stakeholders who stand to benefit from it as part of a wholistic business strategy.

Extracting value from data stores

As storage has gotten cheaper, many companies have begun saving vast quantities of unstructured data such as customer service calls, support request emails, online chats — anything and everything that might someday prove valuable.

“Everyone has been talking about big data and storing it, but nobody has been able to extract value from it and use it,” says Derek Chin, vice president of innovation at Nerdery.

Sentiment analysis could help capture customer insights on a mega scale, he says. “The whole notion is super exciting.”

For example, customers can get annoyed if an agent tries to upsell them. But sentiment analysis can bring up surprising insights about situations in which an upsell is actually helpful, he says.

“Say you have data and WiFi connectivity in your car,” he says. “When your agent can identify that you are almost out of data and can give you an option to buy more to avoid additional service fees, people are happy about that. But if it was a cold offer — ‘I see you’re at a gigabyte a month, and there’s a special deal for two gigabytes a month’ — that would turn people off.”

The key to getting sentiment analysis right, he says, is to understand its limitations, and to be prepared to spend time with it. “In a speech-to-text environment, you’re going to have situations where it doesn’t pick up on sarcasm,” he says. “Those things sort themselves over time as sample sizes get bigger and the AI gets smarter.”

As a result, reacting too quickly to initial data can be dangerous, he says. “You have to let these models train themselves and get enough sample sizes before you make strategic long-term decisions.”

But companies should get started now, he added.

“We are at the very beginning and it is only getting more sophisticated and more impactful down the road, and more of your competitors will start using it,” he says. “Don’t be afraid of it being imperfect now. Start piloting, experimenting with it.”

Going beyond ‘good vs. bad’

Sentiment analysis is already effective in well-defined, simple context with clear-cut outcomes, says Dan Simion, vice president of AI and analytics at Capgemini.

“When it’s in terms of, is it good or bad? That’s when sentiment analysis is working,” he says. “When we start to go into more complicated types of feedback, that’s where there’s still a lot of opportunity for the models to get improved.”

Say, for example, you want to use sentiment analysis to look at photos or videos and tell if people are happy or upset. “Our clients in media and entertainment are trying to understand the sentiment of people watching different shows, and trying to understand what segment of particular shows people find interesting,” he says.

Now that live audiences are back, that could be from analyzing the video feeds from the audience. Or it could be people watching at home, he said, sitting in front of webcams. Traditionally, evaluating responses is a manual process.

But human evaluators are subjective, he says. “You need to have something that is objective, so when you compare the results they start to make sense.”

“And then there’s the problem of scale,” he says. “When you have multiple shows and at the end of the day you want to be consistent, following the same process — that’s somewhere where you need to start to use machines.”

With facial expressions, the sentiment analysis models are still evolving and it’s not even clear yet how to even measure how accurate they are. And even as analyzing the sentiment of facial expressions moves beyond the initial hype phase, there will still be a ways to go, especially for the more nuanced types of facial expressions, before most companies will want to buy in, he says.

“But for companies that are relying on these types of solutions, especially big companies that can afford it and can use it as a competitive advantage, it’s worth it to invest,” Simion says.