Windows 7 Gets Unmasked in Benchmark Testing

Measured by runtime specs and performance benchmarks, Windows 7 M3 looks like Vista, and it runs like Vista. Welcome to Windows Vista R2!

It's here! After months of speculation, Windows 7 was finally unveiled last month at Microsoft's Professional Developers Conference (PDC). Through a series of well-orchestrated keynote presentations and supporting breakout sessions, Microsoft walked conference attendees through the highlights of its new desktop OS: better performance, an improved user experience, and some nifty media-sharing features. Overall, Microsoft's pitch was quite compelling, and the PDC crowd was practically salivating at the chance to play with Microsoft's latest and greatest.

But after the stage props came down, and after the projectors finally went cold, attendees were left with a pre-beta copy of something that looked less like a new OS than the repackaging of an old one. At least that was my impression after I started exploring the Windows 7 M3 (Milestone 3) bits that came on my shiny new 160GB Western Digital USB hard disk (one of the better tchotchkes I've received at a conference). As I reported on my Enterprise Desktop blog, the more I dug into Windows 7, the more I saw an OS that looked and felt like a slightly tweaked version of Windows Vista.

Just what was so new about Microsoft's next Windows, apart from a rejuggled UI? Windows 7 appeared to suck memory like Vista, to consume CPU like Vista, and to have the same consumer focus. How would this product be received by enterprise customers, the vast majority of whom had soundly rejected its predecessor? After all, if Vista wasn't good enough for big business, then surely a Vista-derived encore would meet with a similarly chilly reception.

If any pre-beta software ever called for a close look and benchmark testing, Windows 7 M3 was it. With so many questions to answer, and the fate of Windows in the enterprise hanging in the balance, I rolled up my sleeves and dove in. I started by examining Windows 7's innards -- the kernel and other low-level structures -- then slowly worked my way out to subsystem behavior and application runtime characteristics. Because one of the focal points of Microsoft's keynote presentation was improved performance, I looked for signs that Windows 7 would be faster, more responsive, and less resource-intensive than the bloated Windows Vista.

Note: All the test tools I used for this article are freely available from the Web site. You can also test your current PC for Windows 7 compatibility now, and then monitor Windows 7 performance on your own system when it enters public beta later this year, using InfoWorld's free Windows Sentinel tool.

The view from inside: a minor tweak to Vista As I mentioned in the intro, I began my exploration of Windows 7 by poking around the OS's innards. Using a combination of the Windows Performance Monitor utility and some reference data I'd gathered from Windows Vista and XP, I began comparing the runtime structure and composition of various OS processes and services.

First up was the Windows 7 kernel -- aka the System process. When comparing Windows versions, it's always good to start with the kernel because this is where the most fundamental changes take place. For example, when Microsoft went from Windows 2000 to Windows XP, the System process gained 21 execution threads in its default configuration. Likewise, when Microsoft introduced Windows Vista, the kernel gained 39 execution threads.

In fact, the kernel in each major new version of the Windows OS has spawned a different, typically higher number of threads. So when I examined Windows 7 and found a nearly identical thread count (97 to 100) for the System process, I knew right away that I was dealing with a minor point-type of release, as opposed to a major update or rewrite. This was not "MinWin," the mythical, streamlined new Windows kernel that promised a clean break with the bloated Vista.

Next up: the memory footprint for the kernel. Like the System process, memory footprint tends to vary widely from version to version, with successive releases showing an ever-expanding kernel working set. Based on the thread count values I observed in my previous example, I anticipated that the working set would be similar to Vista's. And in keeping with my initial suspicions, Windows 7 M3's System process indeed consumed a similar amount of memory (3.5MB to 4.5MB). So far, the new OS was looking a lot like Vista.

[ See all of InfoWorld's coverage and analysis of the Windows 7 prerelease in our special report. ]

In fact, as I worked my way through the process lists of the two operating systems, I was struck by the extent of the similarities. Key subsystems, including the Desktop Window Manager (dwm) and Client/Server Runtime (csrss), sported similar thread counts and working set sizes. When viewed side by side in Performance Monitor, Vista and Windows 7 were virtually indistinguishable.

Two of Microsoft's selling points for Windows 7 have been improved performance and lower resource requirements when compared to Vista. However, these sorts of gains traditionally have been difficult to realize without significantly modifying the OS at a kernel level, and the telltale signs were showing that the changes were minor. What would happen if I threw some test workloads at this OS?

Benchmarking Windows 7 Milestone 3 To test Windows 7, I employed the DMS Clarity Studio tools suite -- the same tools I used in March to show that Vista was 40 percent slower than Windows XP. Using a combination of the Clarity Studio's ADO (ActiveX Data Objects), MAPI (Messaging Application Programming Interface), and WMP (Windows Media Player) Stress workload objects, I was able to simulate a complex, multiprocess workload under Windows 7 consisting of client/server database, workflow, and streaming media playback tasks. I used the DMS Clarity Tracker agent to record system and process metrics during the test scenarios. All tests were conducted against a 2GB Core 2 Duo (T7200) laptop PC (the Dell XPS M1710) configured to dual-boot between Windows 7 Ultimate M3 and Windows Vista Ultimate SP1.

A nine-way test scenario, involving three concurrent instances of each workload object, turned in nearly identical average transaction times under Windows 7 M3 and Windows Vista. In fact, the scores were so close -- less than a 5 percent delta (in favor of Vista) on the database tasks, and a roughly 2 percent delta (in favor of Windows 7) on the workflow tasks -- that they fell within what I'd typically consider the margin of error for this sort of test.

An analysis of system and process metrics collected during testing showed both operating systems consuming a similar amount of RAM -- 637MB for Vista and 658MB for Windows 7 -- across all active processes, while the overall thread count showed a modest reduction under Windows 7 (712 versus 810 for Vista). Interestingly, Clarity Studio's process objects for the database workloads (ADO Stress) spent significantly less time (61 percent less) executing in kernel mode under Windows 7 than under Vista, perhaps indicating that some of the code paths related to Jet 4.0 database access have been moved into user mode. This could conceivably explain the modest, 5 percent slowdown of these same workloads under Windows 7. The additional Local Procedure Call overhead of moving portions of MDAC (Microsoft Data Access Components) out of the kernel would most certainly be felt by a time-sensitive, looping transactional workload like ADO Stress.

In a nutshell, Windows 7 M3 is a virtual twin of Vista when it comes to performance. The few minor variations I observed during comparative testing are easily explained away by slight tweaks to the kernel (such as the aforementioned MDAC changes); they certainly don't indicate a significant performance overhaul. More important, these variations pale in comparison with the 40 percent gap in performance I've observed between Vista and Windows XP SP3. From a raw throughput perspective, Windows 7 promises to perform as poorly as its predecessor. "Pre-beta" notwithstanding, the reality is that any hope for closing of the performance gap with Windows XP is unlikely to materialize in Windows 7.

Visual cues and other hints show Vista under the hood Much has been made, by Microsoft and others, of the significant changes to the Explorer UI under Windows 7. The M3 build provided at PDC lacks certain key features -- such as the reengineered task bar --but clearly shows that Microsoft felt the need to do some housecleaning. Printers and other peripherals, including Bluetooth devices, are now managed through a single Control Panel applet, the Device Stage. Some Control Panel items have been eliminated entirely, while others have been shifted around or rolled into neighboring categories (for example, Security and Maintenance).

Overall, the changes are mostly superficial. Even the new Task Bar is simply a twist on the existing Explorer UI model, not to mention a blatant rip-off of the Mac OS X dock. Moreover, none of Windows 7's UI goodness is the result of any architectural changes to the OS. The underpinnings are still clearly based on Vista, which explains why most Vista device drivers and services install without a hitch under Windows 7 M3. Not all Vista drivers behave, however; see the next section for some potential trouble spots.

Otherwise, Windows 7 operates much like Vista. There are subtle visual tweaks here and there, but nothing on the level of the dramatic XP-to-Vista transition. Ironically, Vista users may be more annoyed by the UI changes than users coming from XP. Because the Windows 7 and Vista Aero experiences are so similar, seasoned users of Vista will be more likely to look in the wrong places for common functions. By contrast, XP users won't be burdened with now-outdated Aero navigation skills.

1 2 Page 1
Page 1 of 2
Security vs. innovation: IT's trickiest balancing act