Is Firefox the Most Secure Web Browser?–Part 3

[FULL DISCLOSURE: In addition to being a 20-year Security Guy, I work for Microsoft. While I try hard to focus on objective data, go ahead and assume bias, if you wish, and challenge my analysis with your own comments—you’ll be helping me fulfill my goal of ensuring all sides of security claims are thoroughly examined and rigorously debated in the public view.]

I encourage you to read Part 1 and Part 2 of this series for background and context. As I discussed in Part 1, I challenged Mozilla claims that Firefox “won’t harbor nearly as many security flaws as those that have Microsoft’s Internet Explor” with an Internet Explorer and Firefox Vulnerability Analysis, which resulted in rebuttal from Mozilla’s Mike Shaver (please do read it, so you have his viewpoint).

My main issue with the rebuttal is that rather than address the issue of software vulnerabilities and acknowledging that Writing Secure Code is hard (for everybody in the industry), it takes the approach of changing topics to redirect the conversation away from Firefox and towards Microsoft, asserting that my analysis of security flaws is a poor measure of security. Of course, that ignores the fact that it is a really good measure of specific claims about having low numbers of vulnerabilities.

I still think the issues raised should receive a vigorous and open discussion, today I will dig into one of the two main points raised by Shaver: “You Can Only Count What The Vendor Wants You to See.”

You Can Only Count What the Vendor Wants You to See

Rather, I would say you can count everything that gets disclosed in a broad and public way, by security researchers or vendors. This is at the heart of the theory of Full Disclosure and I am comfortable asserting that most vulnerabilities across the industry in open and closed source models are disclosed by someone other than the vendor or development teams. Vendors do disclose issues, but it is typically a small percentage of the overall set of vulnerabilities.

If that’s true, why is this issue even raised? Probably because of the issue of so-called “silent fixes” and because this was a hotly debated concern a few months before my analysis was released. After studying the discussions quite a bit, I see this as another incarnation of the various interpretations of Full Disclosure and a continuation of the debate between differing camps:

• (position one) Full disclosure requires that full details of a security vulnerability are disclosed to the public, including details of the vulnerability and how to detect and exploit it.
• (position two) The disclosure process and what information is disclosed should be guided by the best efforts to minimize overall risk to users. I lump the “disclose the minimum” crowd in here too.

In general, I hold with the latter (position two) since, when successful, it can result in lower risk for users.

However, for purposes of this article, I want to try to apply the position as articulated by Mozilla and see how they hold up to the standard they assert should be applied to others:

“We count every defect distinctly. We count the ones that Mozilla developers find in-house. We count the things we do to mitigate defects in other pieces of software, including Windows itself and other third-party plugins. We count memory behavior that we think might be exploitable, even if no exploit has ever been demonstrated and the issue in question was found in-house.” (from Counting Still Easy…)

There is the claim. They count every defect distinctly. Now, as far as I know, Mozilla has never actually published any reports or “counts” at all, but I’m not going to quibble about that. Instead, let’s interpret what was really said as that they acknowledge and disclose all of the vulnerabilities that are in Firefox, so that one could count them easily.

Check #1—Does Mozilla “Count” All Vulnerabilities?

Okay, I’ll start with a simple high-level check: A list of the vulnerabilities acknowledged and fixed in a Mozilla Foundation Security Advisory (MFSA). This makes up my “Mozilla has acknowledged the vulnerability” list.

Next, I compiled a list of all vulnerabilities in the NVD that purport to affect Firefox and filtered out any that were on the other list, leaving me with ones that have not (yet) been addressed in a Mozilla Foundation Security Advisory. Further, I removed any that the NVD noted had been marked as DISPUTED and removed any issues that only affected a BETA version of Firefox. I also filtered out ones where it turned out that it actually affected a plug-in rather than Firefox itself. I was left with this list of 44 vulnerabilities which don’t appear to be “counted” in a Mozilla advisory:

CVE-2005-2114, CVE-2005-2395, CVE-2005-4685, CVE-2005-4720, CVE-2005-4809, CVE-2006-0496, CVE-2006-2613, CVE-2006-2788, CVE-2006-4310, CVE-2006-4561, CVE-2006-6585, CVE-2006-6970, CVE-2006-6971, CVE-2007-0801, CVE-2007-0802, CVE-2007-1004, CVE-2007-1084, CVE-2007-1116, CVE-2007-1256, CVE-2007-1736, CVE-2007-1762, CVE-2007-1970, CVE-2007-2162, CVE-2007-2176, CVE-2007-2671, CVE-2007-3072, CVE-2007-3073, CVE-2007-3074, CVE-2007-3827, CVE-2007-4013, CVE-2007-4038, CVE-2007-4041, CVE-2007-4357, CVE-2007-5335, CVE-2007-5415, CVE-2007-5691, CVE-2007-5896, CVE-2007-6715, CVE-2008-0367, CVE-2008-2419, CVE-2008-2786, CVE-2008-3444, CVE-2008-4324, CVE-2008-4723

It may be that fixes are coming in the future, and they would be counted at that point. Even if so, it is going to play havoc with their next “at risk” chart (see Part 2 for more info), since only one of these issues has been publicly disclosed for less than 100 days and only five of them have been public less than a year.

Of course, there is another possibility. It may be that some of the vulnerabilities have already been fixed silently. That seems unlikely, given what Mozilla has said about this:

“It is well known that Microsoft redacts release notes for service packs and bundles fixes, sometimes meaning that you get a single vulnerability “counted” for, say, seven defects repaired. (again, from Counting Still Easy…)

I think I’m safe interpreting that statement to mean that Mozilla thinks silent fixes are bad, bad, bad. So, let’s see if we can do some Critical Thinking and figure this out.

Check #2—Are There Cases of Multiple Mozilla Vulnerabilities Assigned a Single Identifier?

I can easily check to see if any of the CVE identifiers addressed by Mozilla were actually bundled together for that one identifier. I’m not going to be exhaustive, just do some basic searching and see what I find.

• MFSA 2008-69—Lists CVE-2008-5513 as fixed. However text indicates “—one variant could be used—” indicating that there were at least two issues addressed. The SessionStore XSS hazard link indicates four bugzilla entries apply, but I don’t have permission to view them, so I couldn’t get more detail.
• MFSA 2008-68 / CVE-2008-5512—Quoted from the NVD, “—Multiple unspecified vulnerabilities in Mozilla Firefox 3.x—”
• MFSA 2008-52 / CVE-2008-5016—Quoted from the NVD, “—allows remote attackers to cause a denial of service (crash) via multiple vectors—” and has 11 bugzilla cases associated with the single vulnerability identifier.
• CVE-2008-4064, CVE-2008-4063, CVD-2008-4062—Quoted from the NVD, “Multiple unspecified vulnerabilities in Mozilla Firefox—”
• CVE-2008-2798, CVE-2008-2799—Quoted from NVD, “Multiple unspecified vulnerabilities in Mozilla Firefox—” Examining the other links provided, there appear to be three bugzilla cases for the former and four cases for the latter.
• CVE-2006-6505—Quoted from NVD, “Multiple heap-based buffer overflows—”
• CVE-2006-5464—Quoted from NVD, “Multiple unspecified vulnerabilities in the layout engine in Mozilla Firefox—”

That is probably enough examples to make the point clear—there are many more examples that I found, but didn’t dig into. In each of these cases, one vulnerability identifier is being assigned and referenced, when in reality several separate bugs or variants are actually being addressed. I think an accurate description of this might be that from Mozilla, you get a single vulnerability counted for, say, several defects repaired. Sound familiar?

I expect that Mozilla might say that since their entries are searchable and sometimes listed, they are transparent, but it seems like a quibble. One of the main criticisms of “silent fix” behavior that researchers raise is that the individual issues were still easy to find for those that knew how to reverse engineer updates and attackers would “—write exploits for all vulnerabilities regardless of what is in (the) bulletin.” This concern should apply equally here.

The answer to the main question seems clear in any case. Mozilla seems to fix multiple similar issues and variants while bundling the under one identifier.

Check #3—Does Mozilla Silently Fix Issues in New Versions?

There was one final concern which Mozilla raised in Counting Still Easy: “Or maybe you don’t hear about it at all, because it was rolled into SP2 and they didn’t make any noise about it.”

Is it a bad thing to roll lower severity fixes into the less frequent service packs? I have my own thoughts, but I don’t want to digress. From the comment, it seems to me that Mozilla frowns on this behavior in others.

I organized the vulnerability list from above into groups for each of the Firefox versions from Mozilla that has reached end-of-life (EOL). None of these are mentioned in any MFSA that I could find and each of them was publicly disclosed before the product reach EOL. So, what happened with these vulnerabilities? I don’t expect any patches for these versions since they are no longer supported, however, the code base does carry forward to the new version.

Logically, three possibilities occur to me:

• The code was not carried forward to the next version, possibly replaced. This would basically be a fix (intentional or unintentional) via reimplementation.
• The vulnerability is patched, silently. CVE-2007-3072 looks like a possible case of this, for example. There is no MFSA, but the NVD has a link to a bugzilla entry listed in the NVD.
• The vulnerability is still unfixed. CVE-2006-4561 and CVE-2007-1736 appears to be an example of this.

You may want to look into these yourself, maybe these were fixed and I just can’t find an advisory. Silently fixed? Unfixed? Either way, it seems to be the very type of decision that Mozilla used to redirect attention away from the vulnerability counts for the first year of Firefox.

Let me reiterate that I’m not making any claims, I am simply testing the claims made by Mozilla, who asserts that their product is “The Safest Web Browser.”

For the issues I looked into here, I think the results are telling. While it is true that you can only see what the vendor wants you to see, the statement also applies equally to Mozilla, who in spite of their strong words, appear to bundle fixes and silently fix issues where convenient to their development process.

Are any of these decisions unreasonable? Probably not. What it emphasizes to me is something that I’ve heard said in the hallways many times over the past several years—delivering on a commitment to improve security is a hard problem, not just for Microsoft, but for the entire industry.

There is also the other main point raised in in Counting Still Easy… “More fixes means less security?”—which seems to say that fixing more vulnerabilities in 2006 means Mozilla was doing a better job at security. If so, then I would expect to see a downward trend in Firefox vulnerabilities in 2007 and 2008 (since they were just more rigorous in finding and fixing in 2006). If, on the other hand, we find that the trend is not downward, then I think it calls that line of argument into question.

Join me for the next part in this series, where I move away from older 2006 data and arguments and begin to look at how the threat and vulnerability landscape has evolved in 2007 and 2008 and try to discover possible answers to these questions.