How Bug Bounty Programs Bring Big Savings and Better Security

Following in the footsteps of Google, Facebook, PayPal, AT&T and many other enterprises, Microsoft announced last month that it plans to run a bug bounty program of its own. The company will pay up to $100,000 to anyone who finds and can exploit vulnerabilities in its latest version of Windows.

Bug bounty programs are useful to large companies because they encourage hackers to report software vulnerabilities before they’re exploited. The alternative for many hackers is either to publicize vulnerabilities they find for the recognition it brings them among their peers, or attempt to sell the vulnerability—or an exploit based on it—to cybercriminals on the black market or even to government agencies.

Most large enterprises run regular penetration tests and security audits on an occasional basis as a matter of course, but a bug bounty program is an additional measure that encourages a greater number of people to test the system on an ongoing basis. Because bug bounty programs pay only for results, they’re an efficient way for organizations to spend money on security.

Bug bounty programs, however, aren’t easy programs to run: Skilled staff members need to be available to examine all the bug reports that are submitted, validate the bugs and decide how critical they are. There is also an administrative burden in setting up the program and arranging the payment of bounties.

Bug Bounty Programs-as-a-Service

Smaller organizations are beginning to benefit from these programs thanks to an increasing number of companies offering bug bounty programs-as-a-service. Companies such as Bugcrowd, Bugwolf, CrowdSecurify and Hatforce set up and run bug bounty programs on behalf of customers, accepting bug submissions and validating them, as well as making the payouts when necessary.

“We have found a lot of interest in our bug bounty programs because they are much more cost effective than paying penetration testers or consultants,” says Casey Ellis, CEO of San Francisco-based Bugcrowd. “Based on the programs we have done so far we see up to five times as many vulnerabilities found for the same amount of money spent.”

Bugcrowd has signed about 2,200 testers—hackers, security researchers and students from around the world—and groups drawn from this pool can participate in the bug bounty programs the company runs for its clients.

The company actually operates two types of programs: The first type is open-ended, which is similar to ones run by businesses like Google and Facebook. The second type is based on a fixed bounty budget and runs for a set time period, which is similar to a crowdsourced penetration test. Large prizes are awarded for the first few major bugs found, and the remainder of the budget is distributed for more minor bugs that may be discovered.

Bug Bounty Programs More Cost Effective Than Penetration Testing

POLi Payments is an Australia-based online payments system provider which has used Bugcrowd’s bug bounty service. The company’s systems have been penetration-tested by VeriSign and its code has been reviewed by two other organizations, but Jeffery McAlister, POLi Payments’ CEO, points out that the penetration test was carried out to get a report at the end of it.

“Scopes were set, budgets were set, and our technical team was confident that we would get a good report. But unless we came under real attack, we couldn’t be sure how secure we really were,” he says.

POLi Payments decided on a limited period bug bounty program with a fixed budget of AU$5,000 (US$4,650), about half the price of its penetration test. There were 335 people who participated in the program, at least some of whom McAlister believes were former underworld hackers.

“That meant that the attacks they launched were realistic. But I was concerned, at least initially, that the bug bounty program would put our systems on their radar. If they found a hole they might not report it and then come back later.”

But he was reassured by two facts: First, he says, Bugcrowd did seem to have done thorough background checks on all the participants. And because of the size of the test, he expected that most holes would likely be discovered and reported by multiple participants. In fact 80 percent of the bugs that were found were spotted by more than one tester, he says.

McAlister says he was also concerned that the testers might bring down the company’s systems. For that reason, they were instructed not to carry out brute force attacks. “Most of them complied with that, but a few kicked our systems pretty hard and we had to block them. It was certainly a worry.”

The bug bounty program proved to be a more cost effective way of discovering bugs than penetration testing, McAlister says. “The program yielded 38 issues—nothing startling but a lot of minor things—while the last penetration test, which cost more than twice as much, only uncovered a handful of bugs,” he says.

Bigcommerce, an Australian ecommerce solution vendor, also used a bug bounty program. Bigcommerce runs penetration tests annually in order to be PCI compliant, but Peter Whitfield, the company’s engineering director, wanted to take additional security measures.

Bigcommerce’s approach differed from POLI Payments’ in that Bigcommerce decided to set up a copy of its environment in the cloud using Amazon Web Services (AWS), running new components of its software and loaded with dummy data.

“We didn’t want to point the testers at our production system since we couldn’t afford for it to go down, but using the cloud we were able to set up a copy of our environment without having to invest in hardware. We just paid for a week of storage and processing on Amazon,” Whitfield says.

“Our thinking was that if the systems got compromised and the testers got to the underlying data, it wasn’t real data so we didn’t have to worry. We were a little worried about exposing our software to a group of international hackers but that’s what happens in the real world,” he says. “So we decided that it would be better to do it in a controlled way like this so that we could get feedback and deal with any problems they found.”

Penetration Tests vs. Bug Bounty Programs

Whitfield says that the problem with traditional penetration tests is that they are carried out by a small number of people, so the company is reliant on their individual skills. “We have found that different penetration testers find different problems, sometimes problems that have been there for several years and missed in previous tests,” he says. “With the bug bounty program we got a hundred and twenty pairs of eyeballs on our system for a week instead of just one or two pairs for a week.”

After the company’s most recent program, almost forty new bugs were discovered, he says. Compared to last penetration test, only twelve were discovered, he says.

There’s little doubt that penetration testing and code auditing will continue to be an important part of many companies’ security efforts—even if only for regulatory compliance purposes. But thanks to bug bounty-as-a-service offerings, companies like POLi Payments and Bigcommerce can also subject their systems to attack from a large number of hackers in a relatively controlled manner.

“This is definitely now my preferred method of security testing,” says POLi Payments’ McAlister.

Paul Rubens is a technology journalist based in England. Follow everything from CIO.com on Twitter @CIOonline, on Facebook, and on Google +.