We brought all of the products into the production computing environment of a small multinational with six locations in five countries and ran them over the course of four months. Our environment consisted of a distributed range of devices including Linux machines (RedHat and SUSE), Microsoft Windows 2003 systems, VMware ESX servers, Cisco routers, switches (Cisco and HP), intrusion-detection systems, VPN devices (Cisco and Juniper), and firewalls (Cisco PIX and ASA models).

To create an even playing field for testing we first built out a centralized syslog server. This system served two major purposes. First it gave us a single point to "relay" all event data to all SIEM platforms simultaneously. Secondly, it gave us a "clean" data set to perform searches against to validate SIEM findings. For example, if SIEM X failed to detect something we thought it should have, we could go back and look at the raw event data on our syslog server.

To get all of the syslog data distributed we used the open-source syslog-ng package from Balabit IT, with the "spoof source" directive enabled. (We had to re-compile syslog-ng as the native build of the package in SUSE Linux was compiled without this feature.) The spoof-source option is essential to having the SIEM tools properly detect the IP addresses of the log sources.

The native Windows event logging mechanisms threw us a bit of a curveball as Microsoft's servers don't natively use the syslog protocol. Deploying half a dozen Windows event-scrapping agents wasn't attractive or feasible so we opted to deploy the free "Snare" agent from InterSect Alliance on select domain controllers. Snare can scrape native windows event logs and forward them via the syslog protocol, and most of the products we tested supported the format.

Once everything was up and running we began the tuning process, started checking the dashboards on a daily basis, and started building correlation rules and working through our test use-cases.

Learning where features reside in each tool was a bit troubling at first, but over the course of several months we were able to get reasonably familiar with the different approaches the vendors took. For anyone planning on "trying this at home" we have one word of advice: block out some serious time! We spent hundreds of man hours on this project. While time requirements will vary based on complexity, size, use cases tested, and the number of participants, don't plan on it being a short endeavor.