[FULL DISCLOSURE: In addition to being a 20-year Security Guy, I work for Microsoft. While I try hard to
focus on objective data, go ahead and assume bias, if you wish, and challenge my analysis with your own comments—you’ll
be helping me fulfill my goal of ensuring all sides of security claims are thoroughly examined and rigorously debated in the
I encourage you to read Part 1 and Part 2 of this series for background and context.
As I discussed in Part 1, I challenged Mozilla
claims that Firefox “won’t harbor nearly as many security flaws as those that have Microsoft’s Internet Explor” with an
Internet Explorer and Firefox Vulnerability Analysis, which resulted in rebuttal from Mozilla’s Mike Shaver (please do read
it, so you have his viewpoint).
My main issue with the rebuttal is that rather than address the issue of software vulnerabilities and acknowledging that Writing Secure Code is hard (for everybody in the industry), it takes the
approach of changing topics to redirect the conversation away from Firefox and towards Microsoft, asserting that my analysis
of security flaws is a poor measure of security. Of course, that ignores the fact that it is a really good measure of
specific claims about having low numbers of vulnerabilities.
I still think the issues raised should receive a vigorous and open discussion, today I will dig into one of the two main
points raised by Shaver: “You Can Only Count What The Vendor Wants You to See.”
You Can Only Count What the Vendor Wants You to See
Rather, I would say you can count everything that gets disclosed in a broad and public way, by security researchers or
vendors. This is at the heart of the theory of Full Disclosure and I am comfortable asserting that most vulnerabilities across
the industry in open and closed source models are disclosed by someone other than the vendor or development teams. Vendors
do disclose issues, but it is typically a small percentage of the overall set of vulnerabilities.
If that’s true, why is this issue even raised? Probably because of the issue of so-called “silent fixes” and because this
was a hotly debated concern a few months before my analysis was released. After studying the discussions quite a bit, I see
this as another incarnation of the various interpretations of Full Disclosure and a continuation of the debate between
- (position one) Full disclosure requires that full details of a security vulnerability are disclosed to the public,
including details of the vulnerability and how to detect and exploit it.
- (position two) The disclosure process and what information is disclosed should be guided by the best efforts to minimize
overall risk to users. I lump the “disclose the minimum” crowd in here too.
In general, I hold with the latter (position two) since, when successful, it can result in lower risk for users.
However, for purposes of this article, I want to try to apply the position as articulated by Mozilla and see how they hold
up to the standard they assert should be applied to others:
“We count every defect distinctly. We count the ones that Mozilla developers find in-house. We count the things we do to
mitigate defects in other pieces of software, including Windows itself and other third-party plugins. We count memory
behavior that we think might be exploitable, even if no exploit has ever been demonstrated and the issue in question was
found in-house.” (from Counting Still Easy…)
There is the claim. They count every defect distinctly. Now, as far as I know, Mozilla has never actually published any
reports or “counts” at all, but I’m not going to quibble about that. Instead, let’s interpret what was really said as that
they acknowledge and disclose all of the vulnerabilities that are in Firefox, so that one could count them easily.
Check #1—Does Mozilla “Count” All Vulnerabilities?
Okay, I’ll start with a simple high-level check: A list of the vulnerabilities acknowledged and fixed in a Mozilla Foundation
Security Advisory (MFSA). This makes up my “Mozilla has acknowledged the vulnerability” list.
Next, I compiled a list of all vulnerabilities in the NVD that purport to affect Firefox and filtered out any that were on
the other list, leaving me with ones that have not (yet) been addressed in a Mozilla Foundation Security Advisory. Further,
I removed any that the NVD noted had been marked as DISPUTED and removed any issues that only affected a BETA version of
Firefox. I also filtered out ones where it turned out that it actually affected a plug-in rather than Firefox itself. I was
left with this list of 44 vulnerabilities which don’t appear to be “counted” in a Mozilla advisory:
CVE-2005-2114, CVE-2005-2395, CVE-2005-4685, CVE-2005-4720, CVE-2005-4809, CVE-2006-0496, CVE-2006-2613, CVE-2006-2788,
CVE-2006-4310, CVE-2006-4561, CVE-2006-6585, CVE-2006-6970, CVE-2006-6971, CVE-2007-0801, CVE-2007-0802, CVE-2007-1004,
CVE-2007-1084, CVE-2007-1116, CVE-2007-1256, CVE-2007-1736, CVE-2007-1762, CVE-2007-1970, CVE-2007-2162, CVE-2007-2176,
CVE-2007-2671, CVE-2007-3072, CVE-2007-3073, CVE-2007-3074, CVE-2007-3827, CVE-2007-4013, CVE-2007-4038, CVE-2007-4041,
CVE-2007-4357, CVE-2007-5335, CVE-2007-5415, CVE-2007-5691, CVE-2007-5896, CVE-2007-6715, CVE-2008-0367, CVE-2008-2419,
CVE-2008-2786, CVE-2008-3444, CVE-2008-4324, CVE-2008-4723
It may be that fixes are coming in the future, and they would be counted at that point. Even if so, it is going to play
havoc with their next “at risk” chart (see Part 2 for more info), since only one of these issues has been publicly disclosed
for less than 100 days and only five of them have been public less than a year.
Of course, there is another possibility. It may be that some of the vulnerabilities have already been fixed silently. That seems unlikely, given what Mozilla has said about this:
“It is well known that Microsoft redacts release notes for service packs and bundles fixes, sometimes meaning that you get a
single vulnerability “counted” for, say, seven defects repaired. (again, from Counting Still Easy…)
I think I’m safe interpreting that statement to mean that Mozilla thinks silent fixes are bad, bad, bad. So, let’s see if we
can do some Critical Thinking and figure this out.
Check #2—Are There Cases of Multiple Mozilla Vulnerabilities Assigned a Single Identifier?
I can easily check to see if any of the CVE identifiers addressed by Mozilla were actually bundled together for that
one identifier. I’m not going to be exhaustive, just do some basic searching and see what I find.
- MFSA 2008-69—Lists CVE-2008-5513 as fixed. However text indicates “—one variant could be used—” indicating
that there were at least two issues addressed. The SessionStore XSS hazard link indicates four bugzilla entries apply, but I
don’t have permission to view them, so I couldn’t get more detail.
- MFSA 2008-68 / CVE-2008-5512—Quoted from the NVD, “—Multiple unspecified vulnerabilities in Mozilla Firefox
- MFSA 2008-52 / CVE-2008-5016—Quoted from the NVD, “—allows remote attackers to cause a denial of service
(crash) via multiple vectors—” and has 11 bugzilla cases associated with the single vulnerability identifier.
- CVE-2008-4064, CVE-2008-4063, CVD-2008-4062—Quoted from the NVD, “Multiple unspecified vulnerabilities in Mozilla
- CVE-2008-2798, CVE-2008-2799—Quoted from NVD, “Multiple unspecified vulnerabilities in Mozilla Firefox—”
Examining the other links provided, there appear to be three bugzilla cases for the former and four cases for the latter.
- CVE-2006-6505—Quoted from NVD, “Multiple heap-based buffer overflows—”
- CVE-2006-5464—Quoted from NVD, “Multiple unspecified vulnerabilities in the layout engine in Mozilla
That is probably enough examples to make the point clear—there are many more examples that I found, but didn’t dig
into. In each of these cases, one vulnerability identifier is being assigned and referenced, when in reality several
separate bugs or variants are actually being addressed.
I think an accurate description of this might be that from Mozilla, you get a single vulnerability counted for, say, several
defects repaired. Sound familiar?
I expect that Mozilla might say that since their entries are searchable and sometimes listed, they are transparent, but it
seems like a quibble. One of the main criticisms of “silent fix” behavior that researchers raise is that the individual
issues were still easy to find for those that knew how to reverse engineer updates and attackers would “—write exploits
for all vulnerabilities regardless of what is in (the) bulletin.” This concern should apply equally here.
The answer to the main question seems clear in any case. Mozilla seems to fix multiple similar issues and variants while
bundling the under one identifier.
Check #3—Does Mozilla Silently Fix Issues in New Versions?
There was one final concern which Mozilla raised in Counting Still Easy:
“Or maybe you don’t hear about it at all, because it was rolled into SP2 and they didn’t make any noise about it.”
Is it a bad thing to roll lower severity fixes into the less frequent service packs? I have my own thoughts, but I don’t
want to digress. From the comment, it seems to me that Mozilla frowns on this behavior in others.
I organized the vulnerability list from above into groups for each of the Firefox versions from Mozilla that has reached
end-of-life (EOL). None of these are mentioned in any MFSA that I could find and each of them was publicly disclosed before
the product reach EOL. So, what happened with these vulnerabilities? I don’t expect any patches for these versions since
they are no longer supported, however, the code base does carry forward to the new version.
Logically, three possibilities occur to me:
- The code was not carried forward to the next version, possibly replaced. This would basically be a fix (intentional or
unintentional) via reimplementation.
- The vulnerability is patched, silently. CVE-2007-3072 looks like a possible case of this, for example. There is no
MFSA, but the NVD has a link to a bugzilla entry listed in the NVD.
- The vulnerability is still unfixed. CVE-2006-4561 and CVE-2007-1736 appears to be an example of this.
You may want to look into these yourself, maybe these were fixed and I just can’t find an advisory. Silently fixed?
Unfixed? Either way, it seems to be the very type of decision that Mozilla used to redirect attention away from the
vulnerability counts for the first year of Firefox.
Let me reiterate that I’m not making any claims, I am simply testing the claims made by Mozilla, who asserts that their
product is “The Safest Web Browser.”
For the issues I looked into here, I think the results are telling. While it is true that you can only see what
the vendor wants you to see, the statement also applies equally to Mozilla, who in spite of their strong words, appear to
bundle fixes and silently fix issues where convenient to their development process.
Are any of these decisions unreasonable? Probably not. What it emphasizes to me is something that I’ve heard said in the
hallways many times over the past several years—delivering on a commitment to improve security is a hard problem, not
just for Microsoft, but for the entire industry.
There is also the other main point raised in in Counting Still Easy… “More fixes means less security?”—which seems
to say that fixing more vulnerabilities in 2006 means Mozilla was doing a better job at security. If so, then I would expect
to see a downward trend in Firefox vulnerabilities in 2007 and 2008 (since they were just more rigorous in finding and fixing
in 2006). If, on the other hand, we find that the trend is not downward, then I think it calls that line of argument into
Join me for the next part in this series, where I move away from older 2006 data and arguments and begin to look at how the
threat and vulnerability landscape has evolved in 2007 and 2008 and try to discover possible answers to these questions.