In short: if all you have is piece parts, how do you know when the software is working? Or, more important, how can you tell if it's failed? That was the problem that Clear Channel's solution architect Curt Smith and enterprise architect John Szurek stared in the face.

Clear Channel is among the world's largest media companies, specializing in mobile and on-demand entertainment and information services for local communities. The company's businesses include radio (they own 8 percent of the radio stations in the U.S.) and billboards (more formally called outdoor displays).

On the IT side, Clear Channel has a primarily Web-based environment built on Microsoft platforms, with a lot of investment in Web services and cloud computing. "Almost everything we build—certainly our core business applications—are Web apps," explained Smith.

For the last few years, the company has been improving its user interfaces (UIs) to make their applications more common and unified. Concurrently, Smith and Szurek have been doing their best to make the delivery and management monitoring of those applications work more smoothly.

Part of creating a standardized UI is to treat the back-end systems as a commodity engine, Smith explained. Three years ago, they decided to adopt Web services, with the notion of a loosely-coupled world with a common interface. Great. The model works. But...

"From the beginning, some of us—Curt and I at the forefront—realized that there was trouble brewing in paradise," Szurek explained. "These SOAP-type systems that we hate so much turn out to be pretty nice after all. They define what the pieces are." While the business appreciates the agility of loosely-coupled architectures, and programmers were happy with it, said Szurek, "The monitoring people don't want that. The operations people don't want that. And with virtual machines in the data center... there could ultimately be chaos."

Chaos? Hey, isn't loosely-coupled software design supposed to make things simpler? For developers, sure. But for IT departments worried about monitoring a whole enterprise's collection of applications, the difficulty is understanding how it all interconnects and having visibility from one end of an application stream to the other.

You can see if an individual Web service is up or down, explained Smith, but what an operations department cares about is the uptime and performance of the system. If an application is no longer a monolithic construct, but composed of synthetic transactions, how do you define what the application is? Especially when the functionality is workflow-oriented, and thus may gate differently—that is change based on user behavior—depending on how the software is employed.

"We were ill equipped to deal with it," explained Smith. The company's first SOA application was fairly simplistic: a contract commitment product for a line-of-business department. "When that system was first brought up, it broke for 18 hours—and no one knew," said Smith. Because users could fill out the Web form and submit the page, getting an appropriate system response that the data was sent, they were happy. But the data was blocked further down the pipeline, and nothing identified that a failure had occurred. "If you aren't looking at [the transaction] all the way through, you don't see the problem," said Smith. "The interdependency model becomes much, much more complex."

Their newfound awareness led Smith and Szurek to work with Microsoft's Operations Manager (MOM), more formally known today as System Center Operations Manager 2007 and other components of Microsoft's System Center family of products, including the company's Configuration Manager, Data Protection Manager and Virtual Machine Manager (the latter currently is still in beta). The IT organization manages the entire platform—750 servers in the data center, and another 500-600 spread out at other locations—from one central location.

"Any solution for a company is a combination of people, process and technology," said Bob Kelly, Microsoft corporate vice president for Infrastructure Server Marketing. "Often what's forgotten is process."

"Along with MOM came this idea of a 'management pack,'" said Smith, which let the department keep an eye on the health of any particular system. But for the architecture to work, it has to be integrated across the entire application development lifecycle. That starts with the programmers.