Crowdsourcing Offers a Tightly Focused Alternative to Outsourcing

Tom White, a principal program manager for Microsoft Bing has been employing the input of the crowd for several years. The company had been using its own “private crowd” of judges to rate search result relevance to significant success.

The key for Microsoft is that crowd tasks are well-defined and highly targeted. “We developed a very focused expertise with our judges so they could work on one task and one task only,” says White.

What White needed was the option to throw any number of things to the crowd–some new feature or user interface a developer or designer was experimenting with–at a few thousand folks to find out what worked and what didn’t. “It had to do with scale and flexibility,” he explains. “We wanted a larger pool of people we could use for a wide variety of tasks.”

“The offshore approach doesn’t scale as effectively as the crowdsourcing approach. We can access thousands of people in markets it may be difficult for offshore providers to reach,”

Tom White, Microsoft Bing

Getting Agile With the Crowd

Two years ago, White began working with Lionbridge –later adding Appen Butler Hill and ClickWorker for redundancy–to create a larger and more agile crowdsourcing option.

Using the internal platform White’s team had already developed for Bing’s private crowd, a user interface designer with an idea for a better user experience will plug an “a versus b test” in and send it off to the crowd for evaluation.

Bing also employs the crowd to supplement its machine learning processes with some real, live human feedback. The company may take a large set of potential search queries to classify (with the crowd charged to determine, for example, if a query is “adult-oriented”) and use that to inform automated classification.

It’s the kind of work Bing couldn’t simply outsource or offshore because of its high volume, says White. “The offshore approach doesn’t scale as effectively as the crowdsourcing approach. We can access thousands of people in markets it may be difficult for offshore providers to reach,” he says. The crowd is also cheaper and faster than using internal employees, delivering the kind of quick input that developers and designers need.

Besides, White says, using a crowd makes more sense because it’s more reflective of his company’s user base. “The size of the crowd gives us cognitive diversity,” White says. “It’s like we have thousands of people in a room versus dozens.”

Lionbridge, for example, says it has 100,000 crowd workers who have been screened and qualified to complete tasks for clients. And crowdsourcing providers have a strong incentive to source skilled workers, says Martha Crow, Lionbridge’s senior vice president of global enterprise solutions, because customers like Bing pay based on actual output rather than headcount or by the hour.

“We break down the work into a task, we pay our workers based on quality tasks completed, and we get paid by our clients in the same way,” Crow says. “This model completely eliminates the nonproductive time and puts the burden of throughput and quality work on the workers and the service provider.”

Lionbridge’s “crowdforce” includes stay-at-home moms, retirees and recent grads in 100 countries, 90 percent of whom have a college degree, says Crow. The company currently has 25 clients using the crowd for global testing, translation, data management and custom solutions.

Challenges of the Crowd

There are challenges, though. White may have a one-off project for which he needs a crowd in Turkey that he may never use again. “We don’t want our vendor-partner to build out a crowd and direct it to a project that will only take 15 minutes and then not have a need for that market,” says White. “We want to make sure we’re not jerking our vendors around.”

The bigger issue is ensuring that the crowd’s output remains high quality. Much of that is in the vendor’s hands, and White is working with his vendors to better understand their crowdsourcing practices.

Depending on the work, tasks can be dual-sourced or go through a quality assurance process before being delivered to the client, says Crow. White says his crowdsourcing management team spends the majority of its time developing an interface that elicits the best work from the crowd. “That’s the bulk of our investment,” he says.

The model is working, says White. And it’s gaining traction in other areas of the company. Microsoft Research, for example, has begun using the crowdsourcing option as well.

Stephanie Overby is regular contributor to CIO.com’s IT Outsourcing section. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.