Six Questions to Consider Before Building a SOA Testing Team

We interviewed three individuals to get their viewpoints on SOA testing from three different perspectives: trainer, vendor, practitioner. Randy Rice is a leading author, speaker and consultant in the field of software testing and software quality. Randy’s company provides training for SOA testing. Frank Cohen is an author and founder of open source testing software company, Push2Test. Jim Giddings is a practitioner and former SOA road warrior who has worked on several SOA initiatives over the past few years.

What is the biggest change that SOA brings to testing?

RR: Randy states that “accessibility to services” or the “headless nature of services” creates new challenges for testers. Testers will have to test many services without the aid of user interfaces and must perform rigorous testing to validate these services. Randy says that testing services reminds him of the challenges of testing drivers which require test harnesses and testing frameworks.

FC: Frank talks about how services are “always available” which is different from n-tier architectures where “you are in control”. With SOA you have services calling other services and in the world of mashups, testers do not always have knowledge of how these services will be used before deployment. Scale and performance are also high on Frank’s list of challenges.

JG: Jim says you should expect frequent testing cycles much like agile projects and a test driven development methodology should be employed. Jim states that testing governance becomes critical in an SOA. Like his peers, Jim discusses the challenges of the headless nature of services and performance testing.

Are you seeing challenges with testers having the required technical skills to understand SOA?

RR: Randy stresses the importance of strong technical knowledge. He states that testers come from different areas of the enterprise (business, development, etc.) and may not necessarily have the proper skills for SOA. He also stresses the importance of the need for more business orientation which he calls the “forgotten area of testing”. Testers who focus solely on the technology, miss the business process side of the equation. Non-business savvy testers often do not have the right level of access to business people.

FC: Frank points out that testers are starting to code and coders are starting to test. Test driven development has created this shift in roles and responsibilities. If you have testers who cannot code, this may be a challenge. Another point Frank makes is that common record and playback methods do not work for SOA and frameworks are needed. Testers must learn how to work with frameworks which may require scripting and/or coding. Testers must also understand the concepts of statefullness.

JG: Jim mentions that many testers equate SOA to Web Services and do not fully understand the scope of the architecture. Not all testers are technical enough to “dig into the architecture as implemented”. SOA requires more than just black box testing. Jim says SOA requires a combination of black, white, and gray box testing since there is “so much going on at each layer” of the architecture.

What tools (if any) are critical for testing SOA?

RR: Randy speaks about tools that allow for test harnesses and mentions SoapUI and ITKO’s Lisa product as two testing tools that are designed to give testers structural access to services. He says that some existing tools can be extended but not to the level required to effectively test SOA.

FC: Frank states that SOA testing tools should let you do repurposing. For example, the tools should allow the tester to take a developer’s test harness and extend it through functional testing. Frank’s company has built Testmaker Pro, an open source tool that provides testers a framework for repurposing without having to learn new languages like Groovy, Jython, etc.

JG: Jim stresses the importance of performance testing tools and recommends a tool like ITKO’s Lisa to follow proxy services and dynamically detect the path of called services. In some SOA implementations, architects tend to leverage many proxies in front of their services.

What additional risks does SOA introduce to testing and deployments?

RR: With no hesitation, Randy warns of performance penalties and costs. Services are determined at runtime from various locations. This creates unknown integration points that testers do not know in advance. Another key risk is that the organization may not have a robust enough understanding of SOA and what it takes to implement it successfully. This could lead to underfunding, not addressing talent issues, not paying attention to organizational challenges, and many other common mistakes. Randy highlights that a company’s first experience with SOA is a learning experience. I highly recommend that companies seek outside help to bring much needed experience to the projects.

FC: Frank states that “SOA is a methodology not a standard”. SOA is still evolving and the lack of standards creates all kinds of issues. One example that Frank gave me was the fact that there are at least 33 different XML parsers and each has its own performance profile. An SOA architect may define one XML parser as the standard even though this parser may not scale for certain use cases. Frank reiterates that XML parsers have different performance characteristics and there is not a standards body that is providing the proper guidance that architects need to make better informed decisions.

JG: Jim points to rapid change as the big risk in an SOA. I have said in the past that SOA brings simplicity, flexibility, and agility to the business but creates complexity, multiple failure points, and manageability issues for IT. Underneath the promise of SOA is a dangerous world that needs to be managed accordingly. Jim mentions that there are also risks caused by the integration of layer specific components when software is deployed into QA environments. Developers tend to be layer specific and what looked good in unit test may not work well in QA. Jim says that it is likely that the standard smoke test for SOA is a full regression test.

What is the number one challenge for testers tackling SOA for the first time?

RR: Often, testers lack visibility into the overall strategy. Another issue is that in some enterprises, testers have to fight for the tools that they get. They may be challenged to get the additional tools required to effectively test SOA.

FC: Frank reiterated his point that there is “no standards body” and no “start here” guide for tackling SOA. In fact, there is heavy debate on what SOA is. Because of these issues, many training materials and tools are not up to par.

JG: Jim has seen testers struggle with the concepts of a layered based testing approach. When this occurs, testers tend to apply the testing practices that they are comfortable with which may not be sufficient for testing services end to end.

What advice would you give to a testing director/manager when embarking on their first SOA initiative?

RR: Randy recommends to “develop a network of 4-5 people with 1-2 years of SOA experience”. Attending conferences and reading books gives you a foundation of knowledge, but real world lessons learned and advice can be found from practitioners who have been through the battles and experienced real world SOA first hand.

FC: Frank recommends assessing the testing team’s skills and capabilities. Does your team “have the right stuff?” Can you tackle this with internal resources or do you need to outsource some parts of it? Frank feels strongly that you need to have the “right cast of characters” which includes test engineers who “understands all of the interfaces”, know how to rapidly patch an application, and a manager who can “go from requirements to test results”.

JG: Jim’s advice is “Think constant change (like agile)”. He also highly recommends having a strong release management process in place before starting. I can tell you from experience that countless hours and dollars will be wasted if this advice is not taken. Jim also recommends bringing in “an experienced testing architect from planning through certification”. He says that there is not an abundance of SOA testing architects out there. If you can’t find one then find someone who has experience with web services.

Summary

Enterprises who are in the early stages of their SOA initiatives or those contemplating starting a SOA initiative should not forget to address the impacts of SOA as it pertains to testing. Understand the risks, evaluate your internal skills and identify skill gaps, built a network with experienced practitioners, identify the necessary tools and processes that need to be in place, and make sure that the funding takes into account the needs of the testing organization.