Mozilla Patches Fastest. NOT!

[DISCLOSURE: Jeff Jones is a director at Microsoft. You can learn more about him at his technet blog.]

If you take away one thing from reading this, I hope that you take away skepticism. The Mozilla team focuses on security and that is a relatively rare thing in the software industry—and I heartily applaud them for their intent and efforts. I wish more would follow their example. However, I think the vulnerability picture for Firefox may not be quite as rosy as some would wish.

A couple of days ago, Secunia published their Secunia 2008 Report, and one of their tables garnered quite a bit of attention:

• Brian Krebs, Washington Post, Fanning the Flames of the Browser Security Wars
• Brian Prince, eWeek, Security Report Ignites Firefox vs. Internet Explorer Feud
• Ryan Naraine, CNET, Report: Firefox buggier, but issued fixes quicker

In one way, I am quite excited that Mr. Krebs thinks “fixing quickly” is the metric that matters most. I am in the process of updating my H1 2008 report to cover all of 2008, and you may recall, in the previous report Microsoft fixed operating system issues at least twice as fast as other vendors.

On the other hand, I just happen to be in the middle of analyzing Mozilla Firefox vulnerabilities for 2007 and 2008 and results being touted just didn’t ring quite true for what I’ve observed so far in my analysis. So, first, let’s take a look at those three vulnerabilities listed by the Secunia report.

Examine the Three Firefox Report Vulnerabilities

SA32192 Disclosed(2008-10-14) Fixed(2008-11-13) Days Before Patch(30) (This affected Firefox 2)

I was able to identify this one as CVE-2008-4582, addressed in MFSA2008-47. However, though the Secunia Advisory did post on 10/14/2008, my findings are that LIUDIEYU disclosed this issue publicly in his blog entry on 9/27/08. That would be an additional 17 days.

SA32040 Disclosed(2008-10-01) Fixed(2008-12-26) Days Before Patch(86) (This affected Firefox 3)

I was able to identify this one as CVE-2008-4324. Though the Secunia advisory lists “vendor patch” under solution, a status change made on 12-26-2008, Mozilla has not released a security advisory that mentions either the Secunia advisory or the vulnerability identifier. SA32040 advises that upgrading to Firefox 3.0.4 or later will fix the issue. I am going to assume that 3.0.4 did silently fix the issue, however that version was released on 11/12/08. That means days before patch should be 42, rather than 86.

SA28622 Disclosed(2008-1-24) Fixed(2008-2-12) Days Before Patch(15) (This affected Firefox 2)

I was able to identify this one as CVE-2008-0418, addressed in MFSA2008-05. SA28622 posted on 1/24/08, but I place the first public disclosure on 1/19/08 athiredhacker. On the other hand, MFSA2008-05 appears to have published on 2/7/08. Taking into account both the earlier disclosure date and the earlier fix date, I believe the days before patch were 19.

After examining these three, I can say that I am very glad that there weren’t more! It seems like Secunia is using Secunia advisory publication dates as the date for “public disclosure”, though as we see in two of the cases, the vulnerability information was public even earlier. Let me point out that there may be similar differences in the numbers concerning Microsoft.

And yes, I am also aware that my findings actually improved the Firefox numbers 😉 I’m not done yet.

What About Earlier Disclosures?

Perhaps more important than any of this, though, is that the Secunia report specifically limited scope to vulnerabilities disclosed during 2008. I certainly have no issue with that—you have to limit scope to be able to make the work finite. However, it absolutely affects results when folks start drawing conclusions about who fixes disclosed issues the quickest.

If a user (or reporter) wanted to draw conclusions about which vendor fixed vulnerabilities the fastest, I do not understand how one can realistically excludes issues that were disclosed before 2008 and fixed later. So here is my question for those that are really interested in answering the question of how quickly Mozilla fixes vulnerabilities. What is the average if you include these below (feel free to validate them yourselves to assure yourself that they apply)? Also note that I am only listing ones rated high severity in the NVD or Critical in a Mozilla advisory. I also limited my search to Firefox 2 vulnerabilities.

• CVE-2007-1736, disclosed 3/28/2007, no MFSA after 631 days (352 in 2008) at product end-of-life
• CVE-2007-2162, disclosed 4/18/07, no MFSA after 610 days (352 in 2008) at product end-of-life
• CVE-2007-2671, disclosed 5/1/2007, no MFSA after 597 days (352 in 2008) at product end-of-life
• CVE-2007-3072, disclosed 6/4/2007, no MFSA after 563 days (352 in 2008) at product end-of-life
• CVE-2007-3073, disclosed 6/4/2007, no MFSA after 563 days (352 in 2008) at product end-of-life
• appears to be silently fixed in FF3.0 on 9/30/08—maybe in FF2, can’t tell
• CVE-2007-5896, disclosed 11/2/2007, no MFSA after 412 days (352 in 2008) at product end-of-life

I’m not going to do the math, but if you include these six Firefox 2 issues in with the three from the Secunia report, I’m pretty sure the number will be closer to 352 than it will be to zero.

Of course, it may be that some of these issues above were silently fixed by Mozilla. I wouldn’t mind at all if they came out and confirmed my earlier analysis that they may be doing this. It would bring the numbers down a little.

What Else Was Excluded?

Anybody remember this headline? Code execution vulnerability found in Firefox 3.0 | Zero Day | ZDNet.com . There were several variations on it at the time of Firefox 3 release back in June, in reference to:

• Tippingpoint ZDI post, and Mozilla even responded
• https://blog.mozilla.com/security/2008/06/18/new-security-issue-under-investigation/
• CVE-2008-2786 was assigned to this vulnerability.

As far as I can tell, this issue is still “under investigation.” This wouldn’t have been included in Secunia calculations since details have not yet been published.

Final Thoughts

I would like to conclude by considering Brian’s opinion about the metric that matters most:

In trying to draw conclusions from the data, though, I hope readers will look past the sheer numbers of security holes that each browser maker fixed this past year, to the metric that in my opinion matters most: How long did it take each browser maker to address security flaws once those vendors knew about them?

As it turns out, I think how quickly a vendor fixes issues is a very important issue too. Of course, I would probably stop short of saying that it is the metric that “matter most” and wouldn’t ask readers to look past other metrics that measure different things. I see this mistake (and yes I do consider it a mistake) happen a lot when people try to identify a single metric that represent “the security of” a product. Very briefly, think about the two things Brian refers to here:

Vendor fix time. In his words, the metric that matters most. This measures how quickly a vendor can get a fix to customers once they know about an issue. It doesn’t reflect security quality of the software, for example. Take an extreme example and imagine:

• a product that has a flaw revealed on each day of the year and each is patched 10 days later, and
• a product that has one flaw per year and it takes 30 days to fix .

This metric measures resource, policy and processes. It directly influences, in combination with other factors, exposure time for customers. It measures nothing to do with the inherent security quality or architecture of the software.

Volume of security holes. This metric used to be a very important metric to Mozilla. Remember when Mozilla Executive proudly claimed that Firefox “won’t harbor nearly as many security flaws as those that have Microsoft’s Internet Explorer“? It seems like Firefox advocates now want to de-emphasize the metric. It seems to me that, unlike the other metric, this one actually relates to the success of the team in attempting to “write secure code.”

And of course, there are lots of other security metrics that are important as well. I personally think manageability and ability to enforce security policy are two key ones, for example. Back to the topic at hand.

I am going to be publishing my own full analysis of Mozilla Firefox vulnerabilities and exposure during 2007 and 2008 soon, which is why I was able to pull the above examples so quickly. In that paper, I’ll include what I’ve found here, plus provide updated exposure charts and I hope you’ll come back to look at those details.