If there’s a poster child for the challenges facing open source security, it may be Werner Koch, the German developer who wrote and for the last 18 years has toiled to maintain Gnu Privacy Guard (GnuPG), a pillar of the open source software ecosystem.
Since its first production release in 1999, GnuPG has become one of the most widely used open source security tools in the world, protecting the email communication of everyone from government officials to Edward Snowden.
Yet Koch found himself struggling to make ends meet in recent years. The estimated $25,000 he collected on average in annual donations since 2001 weren’t enough to support his efforts. As reported by Pro Publica, the 53-year-old was close to throwing in the towel on GnuPG when Edward Snowden’s NSA revelations shocked the world, convincing Koch to soldier on. "I'm too idealistic," he said.
The story has a happy ending. After the ProPublica story broke, donors from around the world rushed to support Koch. He easily surpassed the $137,000 fundraising goal he had set to support his work, enabling him to hire a part-time developer. Koch was awarded a one-time grant of $60,000 from the Linux Foundation’s Core Infrastructure Initiative. Facebook and the online payment processor Stripe each pledged $50,000 a year to Koch’s project.
Underfunded projects, as GnuPG was until recently, form part of a vast open source ecosystem unprecedented in scale. Widespread reuse of open source code fuels today’s surging technology development, but the sheer volume of that code discourages security vetting. Only recently have we begun to confront the problem, often on the heels of security breaches that embarrass the industry into action.
Will code for food
The conditions that left Koch high and dry for years are not unusual.
After Google researcher Neel Mehta uncovered Heartbleed, a serious and remotely exploitable vulnerability in a component of OpenSSL, the software community was shocked to learn that the project was largely the responsibility of what Jim Zemlin, executive director of the Linux Foundation, referred to as “two guys named Steve.” Dr. Stephen Henson and Steve Marquess labored part-time to keep the code up to date, compensated by a few thousand dollars a year in voluntary contributions.
Technology vendors who rely on open source were quick to swoop in and set the OpenSSL project to rights. The Core Infrastructure Initiative that gave GnuPG’s creator a $60,000 grant was established months earlier to help fund the work of Henson and others on OpenSSL. Financial support is provided by such Silicon Valley giants as Amazon, Adobe, Cisco, Facebook, and Google.
A thousand eyes blink
Heartbleed wasn’t the first killer open source bug. For example, the Apache Struts vulnerability predated it by almost a year and was at least as serious.
But thanks to the media frenzy, Heartbleed may have permanently discredited Eric Raymond’s famous adage about open source quality: “Given enough eyeballs, all bugs are shallow.” Most security professionals say the notion was always more aspirational than descriptive.
“I never liked the ‘with many eyeballs’ notion,” says Joshua Corman, CTO at the firm Sonatype. “Just saying there are ‘many eyeballs’ doesn’t mean that those eyeballs are motivated or qualified to look to find security vulnerabilities.”
Open source’s “many eyes” assurance served mostly to gloss over a weakness in the open source ecosystem, implying an atmosphere of constant vigilance where none existed, says Bill Weinberg, senior director of open source strategy at the firm Black Duck Software.
“With Shellshock, you didn’t have many eyes,” Weinberg says, referring to critical vulnerability that was found lurking in code for Bash in 2014. “That code was considered to be well vetted, but it turned out that it wasn’t well curated, because everyone assumed it was well vetted.”
While we might like to assume that the integrity of open source code is high, data from Sonatype suggests the opposite. An analysis by the company of open source components in its managed codebase found that known vulnerabilities in open source components were remediated only 41 percent of the time, Corman wrote in the Usenix journal ;login. For the issues that were fixed, the mean time to remediate them was a whopping 390 days.
But even talking about “open source” apart from commercial, proprietary software is misleading. While a line may have once separated open source and proprietary software projects, most modern applications are assemblages of third-party software components, many of them open source, Corman said.
Getting serious about code-level security
What’s the proper response? For better or worse, the answer is largely cultural, says Katie Moussouris, chief policy officer at the firm HackerOne and a former senior security strategist at Microsoft. “We need to build a security mind-set. This is important to every software project -- open source or not.”
Moussouris’ company provides a Web-based platform for coordinating vulnerability disclosures, including through bug bounty programs. She notes that HackerOne already sponsors bug bounties for a wide range of open source projects, including PHP, Ruby on Rails, Python, and OpenSSL, providing compensation for reports of vulnerabilities.
Open source projects need to take a more serious and systematic approach to security, she says: “You have to at least try to build security in.”
Sonatype’s Corman advocates a more rigorous solution: a supply chain akin to those used by manufacturers that delivers both high quality and accountability.
Using the analogy of a Ford manufacturing line, Corman notes that the company knows the provenance of each part that goes into its finished automobiles. Problems can be traced back to specific suppliers, facilities, and even production runs.
In modern software development organizations, however, no such system exists. Almost every commercial software application makes use of open source and proprietary components sold by third-party firms, but software firms may have a cursory notion of the quality and origin of all that code. Often, the reach and impact of vulnerabilities become known only after disaster strikes.
In the case of Shellshock, for example, the code in question dated back to 1989 and impacted a wide range of applications -- from CGI-based Web servers to Qmail email servers to certain DHCP clients. Attacks against the vulnerability began within hours of its disclosure.