To be effective with the time we have, we want to test the most powerful test scenarios, tests that check many, many different variables at one time; we call these "soap opera tests." For example, in an insurance system, that might be a man who starts a business with the wife having primary coverage. Then she has a baby and leaves the husband two weeks later, keeping the baby and herself on the policy. Then the man's long-lost son, who is in college, returns, and the (now ex-)wife agrees to purchase a policy for the son. Two days before the son turns 21 and becomes ineligible for the policy, he has a heart attack. The tests are: does the insurance policy pay out? To whom?

Sounds a bit like a soap opera, doesn't it? Soap opera tests stress multiple business rules at the same time, making them very powerful. The logic is very simple: If basic testing and the soap opera tests pass, the general case will probably do just fine.

Live Data Testing

John McConda, a consultant at Moser Consulting, points out that many testing efforts are naive in that they ignore the full end-game scenario. After, all, the ultimate test of a service-oriented architecture is "orchestration:" all of the pieces working together under a realistic load, such as a monthly billing cycle. According to McConda, "It seems to me that what you really want is for all these systems to work together; I've seen test plans that checked each component but failed to consider if the whole was going to work together with real data."

The simplest way to do this is to have a copy production, perhaps with unique identifiers like social security number removed, and perform the largest operations the system will have, such as bill batches, claims batches and so.

A Few Lessons Learned

SOA is more than a technique; it is a design philosophy. Another core design philosophy is incremental and evolving delivery, instead of big bang up front. In other words, the easiest way to have a huge, messy SOA is to develop each system independently and then to put off integration until the end.

Instead, allow your SOA project to start with two or three components that need to work together. Integrate them successfully, deploy to production, and then add piecemeal.

Second, be sure to test multiple passes of the software. I remember one project where the account balances were stored on the first pass for every transaction. I was brought in to diagnose the bugs. Of course, the lookup was failing; it had never been tested.