[FULL DISCLOSURE: In addition to being a 20-year Security Guy, I work for Microsoft. While I try hard to \n\nfocus on objective data, go ahead and assume bias, if you wish, and challenge my analysis with your own comments\u2014you'll \n\nbe helping me fulfill my goal of ensuring all sides of security claims are thoroughly examined and rigorously debated in the \n\npublic view.]\nI encourage you to read Part 1 and Part 2 of this series for background and context.\nAs I discussed in Part 1, I challenged Mozilla \n\nclaims that Firefox "won't harbor nearly as many security flaws as those that have Microsoft's Internet Explor" with an \n\nInternet Explorer and Firefox Vulnerability Analysis, which resulted in rebuttal from Mozilla's Mike Shaver (please do read \n\nit, so you have his viewpoint).\nMy main issue with the rebuttal is that rather than address the issue of software vulnerabilities and acknowledging that Writing Secure Code is hard (for everybody in the industry), it takes the \n\napproach of changing topics to redirect the conversation away from Firefox and towards Microsoft, asserting that my analysis \n\nof security flaws is a poor measure of security. Of course, that ignores the fact that it is a really good measure of \n\nspecific claims about having low numbers of vulnerabilities.\nI still think the issues raised should receive a vigorous and open discussion, today I will dig into one of the two main \n\npoints raised by Shaver: "You Can Only Count What The Vendor Wants You to See."You Can Only Count What the Vendor Wants You to SeeRather, I would say you can count everything that gets disclosed in a broad and public way, by security researchers or \n\nvendors. This is at the heart of the theory of Full Disclosure and I am comfortable asserting that most vulnerabilities across \n\nthe industry in open and closed source models are disclosed by someone other than the vendor or development teams. Vendors \n\ndo disclose issues, but it is typically a small percentage of the overall set of vulnerabilities.\nIf that's true, why is this issue even raised? Probably because of the issue of so-called "silent fixes" and because this \n\nwas a hotly debated concern a few months before my analysis was released. After studying the discussions quite a bit, I see \n\nthis as another incarnation of the various interpretations of Full Disclosure and a continuation of the debate between \n\ndiffering camps:\n\n(position one) Full disclosure requires that full details of a security vulnerability are disclosed to the public, \n\nincluding details of the vulnerability and how to detect and exploit it.\n(position two) The disclosure process and what information is disclosed should be guided by the best efforts to minimize \n\noverall risk to users. I lump the "disclose the minimum" crowd in here too. \nIn general, I hold with the latter (position two) since, when successful, it can result in lower risk for users.\n\n\nHowever, for purposes of this article, I want to try to apply the position as articulated by Mozilla and see how they hold \n\nup to the standard they assert should be applied to others:\n"We count every defect distinctly. We count the ones that Mozilla developers find in-house. We count the things we do to \n\nmitigate defects in other pieces of software, including Windows itself and other third-party plugins. We count memory \n\nbehavior that we think might be exploitable, even if no exploit has ever been demonstrated and the issue in question was \n\nfound in-house." (from Counting Still Easy...)\nThere is the claim. They count every defect distinctly. Now, as far as I know, Mozilla has never actually published any \n\nreports or "counts" at all, but I'm not going to quibble about that. Instead, let's interpret what was really said as that \n\nthey acknowledge and disclose all of the vulnerabilities that are in Firefox, so that one could count them easily.\nCheck #1\u2014Does Mozilla "Count" All Vulnerabilities?\nOkay, I'll start with a simple high-level check: A list of the vulnerabilities acknowledged and fixed in a Mozilla Foundation \n\nSecurity Advisory (MFSA). This makes up my "Mozilla has acknowledged the vulnerability" list.\nNext, I compiled a list of all vulnerabilities in the NVD that purport to affect Firefox and filtered out any that were on \n\nthe other list, leaving me with ones that have not (yet) been addressed in a Mozilla Foundation Security Advisory. Further, \n\nI removed any that the NVD noted had been marked as DISPUTED and removed any issues that only affected a BETA version of \n\nFirefox. I also filtered out ones where it turned out that it actually affected a plug-in rather than Firefox itself. I was \n\nleft with this list of 44 vulnerabilities which don't appear to be "counted" in a Mozilla advisory:\nCVE-2005-2114, CVE-2005-2395, CVE-2005-4685, CVE-2005-4720, CVE-2005-4809, CVE-2006-0496, CVE-2006-2613, CVE-2006-2788, \n\nCVE-2006-4310, CVE-2006-4561, CVE-2006-6585, CVE-2006-6970, CVE-2006-6971, CVE-2007-0801, CVE-2007-0802, CVE-2007-1004, \n\nCVE-2007-1084, CVE-2007-1116, CVE-2007-1256, CVE-2007-1736, CVE-2007-1762, CVE-2007-1970, CVE-2007-2162, CVE-2007-2176, \n\nCVE-2007-2671, CVE-2007-3072, CVE-2007-3073, CVE-2007-3074, CVE-2007-3827, CVE-2007-4013, CVE-2007-4038, CVE-2007-4041, \n\nCVE-2007-4357, CVE-2007-5335, CVE-2007-5415, CVE-2007-5691, CVE-2007-5896, CVE-2007-6715, CVE-2008-0367, CVE-2008-2419, \n\nCVE-2008-2786, CVE-2008-3444, CVE-2008-4324, CVE-2008-4723\n\nIt may be that fixes are coming in the future, and they would be counted at that point. Even if so, it is going to play \n\nhavoc with their next "at risk" chart (see Part 2 for more info), since only one of these issues has been publicly disclosed \n\nfor less than 100 days and only five of them have been public less than a year.\n\n\nOf course, there is another possibility. It may be that some of the vulnerabilities have already been fixed silently. That seems unlikely, given what Mozilla has said about this:\n"It is well known that Microsoft redacts release notes for service packs and bundles fixes, sometimes meaning that you get a \n\nsingle vulnerability "counted" for, say, seven defects repaired. (again, from Counting Still Easy...)\nI think I'm safe interpreting that statement to mean that Mozilla thinks silent fixes are bad, bad, bad. So, let's see if we \n\ncan do some Critical Thinking and figure this out.\nCheck #2\u2014Are There Cases of Multiple Mozilla Vulnerabilities Assigned a Single Identifier?\nI can easily check to see if any of the CVE identifiers addressed by Mozilla were actually bundled together for that \n\none identifier. I'm not going to be exhaustive, just do some basic searching and see what I find.\n\nMFSA 2008-69\u2014Lists CVE-2008-5513 as fixed. However text indicates "\u2014one variant could be used\u2014" indicating \n\nthat there were at least two issues addressed. The SessionStore XSS hazard link indicates four bugzilla entries apply, but I \n\ndon't have permission to view them, so I couldn't get more detail. \nMFSA 2008-68 \/ CVE-2008-5512\u2014Quoted from the NVD, "\u2014Multiple unspecified vulnerabilities in Mozilla Firefox \n\n3.x\u2014"\nMFSA 2008-52 \/ CVE-2008-5016\u2014Quoted from the NVD, "\u2014allows remote attackers to cause a denial of service \n\n(crash) via multiple vectors\u2014" and has 11 bugzilla cases associated with the single vulnerability identifier. \nCVE-2008-4064, CVE-2008-4063, CVD-2008-4062\u2014Quoted from the NVD, "Multiple unspecified vulnerabilities in Mozilla \n\nFirefox\u2014" \nCVE-2008-2798, CVE-2008-2799\u2014Quoted from NVD, "Multiple unspecified vulnerabilities in Mozilla Firefox\u2014" \n\nExamining the other links provided, there appear to be three bugzilla cases for the former and four cases for the latter.\nCVE-2006-6505\u2014Quoted from NVD, "Multiple heap-based buffer overflows\u2014"\nCVE-2006-5464\u2014Quoted from NVD, "Multiple unspecified vulnerabilities in the layout engine in Mozilla \n\nFirefox\u2014"\nThat is probably enough examples to make the point clear\u2014there are many more examples that I found, but didn't dig \n\ninto. In each of these cases, one vulnerability identifier is being assigned and referenced, when in reality several \n\nseparate bugs or variants are actually being addressed. \nI think an accurate description of this might be that from Mozilla, you get a single vulnerability counted for, say, several \n\ndefects repaired. Sound familiar?\nI expect that Mozilla might say that since their entries are searchable and sometimes listed, they are transparent, but it \n\nseems like a quibble. One of the main criticisms of "silent fix" behavior that researchers raise is that the individual \n\nissues were still easy to find for those that knew how to reverse engineer updates and attackers would "\u2014write exploits \n\nfor all vulnerabilities regardless of what is in (the) bulletin." This concern should apply equally here.\nThe answer to the main question seems clear in any case. Mozilla seems to fix multiple similar issues and variants while \n\nbundling the under one identifier. \nCheck #3\u2014Does Mozilla Silently Fix Issues in New Versions?\nThere was one final concern which Mozilla raised in Counting Still Easy:\n"Or maybe you don't hear about it at all, because it was rolled into SP2 and they didn't make any noise about it."\nIs it a bad thing to roll lower severity fixes into the less frequent service packs? I have my own thoughts, but I don't \n\nwant to digress. From the comment, it seems to me that Mozilla frowns on this behavior in others.\nI organized the vulnerability list from above into groups for each of the Firefox versions from Mozilla that has reached \n\nend-of-life (EOL). None of these are mentioned in any MFSA that I could find and each of them was publicly disclosed before \n\nthe product reach EOL. So, what happened with these vulnerabilities? I don't expect any patches for these versions since \n\nthey are no longer supported, however, the code base does carry forward to the new version. \n \nFirefox 1.0\nFirefox 1.5 \nFirefox 2.0\n\n\nCVE-2005-2114\nCVE-2005-4685\nCVE-2006-6585\n\n\nCVE-2005-2395\nCVE-2006-2613\nCVE-2006-6970\n\n\nCVE-2005-4685\nCVE-2006-2788\nCVE-2006-6971\n\n\nCVE-2005-4720\nCVE-2006-4310\nCVE-2007-0802\n\n\nCVE-2005-4809\nCVE-2006-4561\nCVE-2007-1004\n\n\nCVE-2006-0496\nCVE-2007-0801\nCVE-2007-1084\n\n\nCVE-2006-2788\nCVE-2007-1084\nCVE-2007-1116\n\n\nCVE-2007-1084\n\nCVE-2007-1256\n\n\n\n\nCVE-2007-1736\n\n\n\n\nCVE-2007-1762\n\n\n\n\nCVE-2007-2162\n\n\n\n\nCVE-2007-2671\n\n\n\n\nCVE-2007-3072\n\n\n\n\nCVE-2007-3073\n\n\n\n\nCVE-2007-3074\n\n\n\n\nCVE-2007-4038\n\n\n\n\nCVE-2007-4041\n\n\n\n\nCVE-2007-4357\n\n\n\n\nCVE-2007-5335\n\n\n\n\nCVE-2007-5415\n\n\n\n\nCVE-2007-5691\n\n\n\n\nCVE-2007-5896\n\n\n\n\nCVE-2008-0367\n\n\n\n\nCVE-2008-2419\n\n\n\n\nCVE-2008-2786\n\n\nLogically, three possibilities occur to me:\n\nThe code was not carried forward to the next version, possibly replaced. This would basically be a fix (intentional or \n\nunintentional) via reimplementation. \n\nThe vulnerability is patched, silently. CVE-2007-3072 looks like a possible case of this, for example. There is no \n\nMFSA, but the NVD has a link to a bugzilla entry listed in the NVD. \nThe vulnerability is still unfixed. CVE-2006-4561 and CVE-2007-1736 appears to be an example of this.\nYou may want to look into these yourself, maybe these were fixed and I just can't find an advisory. Silently fixed? \n\nUnfixed? Either way, it seems to be the very type of decision that Mozilla used to redirect attention away from the \n\nvulnerability counts for the first year of Firefox.\nLet me reiterate that I'm not making any claims, I am simply testing the claims made by Mozilla, who asserts that their \n\nproduct is "The Safest Web Browser."\nFor the issues I looked into here, I think the results are telling. While it is true that you can only see what \n\nthe vendor wants you to see, the statement also applies equally to Mozilla, who in spite of their strong words, appear to \n\nbundle fixes and silently fix issues where convenient to their development process.\nAre any of these decisions unreasonable? Probably not. What it emphasizes to me is something that I've heard said in the \n\nhallways many times over the past several years\u2014delivering on a commitment to improve security is a hard problem, not \n\njust for Microsoft, but for the entire industry.\nThere is also the other main point raised in in Counting Still Easy... "More fixes means less security?"\u2014which seems \n\nto say that fixing more vulnerabilities in 2006 means Mozilla was doing a better job at security. If so, then I would expect \n\nto see a downward trend in Firefox vulnerabilities in 2007 and 2008 (since they were just more rigorous in finding and fixing \n\nin 2006). If, on the other hand, we find that the trend is not downward, then I think it calls that line of argument into \n\nquestion. \nJoin me for the next part in this series, where I move away from older 2006 data and arguments and begin to look at how the \n\nthreat and vulnerability landscape has evolved in 2007 and 2008 and try to discover possible answers to these questions.