Some development teams are building a service-oriented architecture the old fashioned way: fast. Leapfrog Enterprises director of Systems Infrastructure Eugene Ciurana describes seat-of-the-pants SOA design in a virtual environment.
By Rich Levin
Cue the Western music, because the coding cowboy hasn’t ridden off into the sunset just yet. And as this case study demonstrates, perhaps we should make sure he doesn’t.
In fact, despite all the industry big-think about how SOA requires strong governance, months of front-end BPA and modeling, baked-in standards (WS* this-and-that), ESBs and registries or repositories and mile-high SOA-savvy integrated vendor stacks, some development teams are building a SOA the old fashioned way: fast.
And it’s working. That’s the case at Leapfrog, one of the world’s most successful makers of educational electronic toys. The company is nearing the end of an ambitious effort to make its consumer product lineup Web-savvy.
When Leapfrog’s new devices connect to the Internet, they are transformed from toys to solutions: the front-end of a sophisticated SOA that integrates embedded devices, legacy systems, e-commerce, and a bevy of reusable services that power features, functionality and ultimately, sales.
But you won’t find a bunch of business analysts and programmers sitting in a room, mapping out requirements and design. In fact, you won’t even find a room. The system is being built virtually, using what could be called a modified Agile development method. The developers are scattered around the globe, communicating with e-mail, Microsoft Word documents, and Internet relay chat (IRC).
The programmers can use any tools, techniques, or technologies they are comfortable with, as long as the resulting components meet spec. And the team leader, who you’ll meet here, is maximizing productivity and quality by hiring consultants who are key players within the open-source community; the people responsible for developing the open-source platforms being used.
Can you build in 18 months what some organizations can only design in 18 months? Eugene Ciurana, director of Systems Infrastructure for Leapfrog Enterprises, is proving you can—and for just 30 percent of what it would cost, he says, had Leapfrog used commercial products and a more structured approach.
This interview was originally planned as a podcast; however, the audio quality was poor due to an error on my part. As a result, I have had the interview transcribed, and present it here.
CIO: Why don’t you tell folks who you are, your title, the company you work for, and then we’ll jump into the conversation?
Ciurana: Sure. My name is Eugene Ciurana. I am the director of Systems Infrastructure for Leapfrog Enterprises.
CIO: And tell us about Leapfrog. What’s the company do?
Ciurana:Leapfrog is the largest independent producer of educational products in the U.S. We have a presence in 35 countries, and we’re a public company. Most people who have kids between the ages of 4 and 10 probably have heard of us. Basically, our mission statement is “helping kids to learn how to read.”
CIO: Great. And tell us a little bit about what you do as the director of Systems Infrastructure.
Ciurana: My charge is to come up with and execute direction for all Web systems that Leapfrog Enterprises will roll out, and which will have a lifetime of about five years. We’re on the first year of our five-year plan, and every single [consumer] product that we’ll probably be launching this year is Web enabled.
Basically, your kids can play games, and learn things like math and reading and writing and spelling, and all the 3Rs, and so on. And then you can connect the device that they’re playing with to the Web via your computer, and download software titles from there. Or you can track the progress of your child in terms of how well he or she is learning whatever topic you’re working on, using something we call the Learning Desk, which is a Web-based console.
You can come in and check on your kid’s program and see how she is doing, and eventually, you may be able to share that information with her teachers. So we have this infrastructure that has to support many of the devices that talk to each other over the Internet, through our servers. Plus we’ve got partners that provide us with other services, like community Web sites and so on. And we have to make all of that look like a seamless experience to the user.
CIO:What was the IT environment, the infrastructure, like when you came onboard, and what has driven—obviously the requirement is fundamentally driving this—but what drove the decision to bring somebody like you in? And what moved you in the direction of a service-oriented architecture?
Ciurana: Leapfrog Enterprises went through a major restructuring sometime in late 2006. At that point the company went from “We’re only making devices, hand-held consoles, plushy toys, and so on and so forth” to “We’re going to things that are Internet-ready.”
We want to be able to leverage the Internet to share information between teachers, parents, children, etc. That became the value proposition for the company. In 2007, we decided to run with the IT side of the project once the business side was set up. Management hired a bunch of us from other universes, where we had been doing SOA and things like that, and they said, “Great, we need to build this huge system. How do we go about it?”
When I came in, we started communicating with a lot of people in the open-source community. We told them what we wanted to do, and we started looking at what kinds of technology we needed to build the system. The Leapfrog business plan called for a complete revamping of the Internet infrastructure in a period of roughly 18 months. Now, we have an established IT infrastructure. They do mail, they do servers, they do the back-end, they do Oracle stuff, and so on.
Our charge was not to tinker with the IT infrastructure, but rather, to [work] with a systems infrastructure that supports our electronic devices. When we saw the business plan, first we said, “Holy cow! That’s a lot of work for 18 months!” And we started trying to figure out how we can get where we need to be the fastest.
Complicating matters is the fact that we have two major projects going on in parallel. One is the Leapfrog.com e-commerce site, which generates quite a bit of money for the company annually. We need to make sure that that’s working very well.
The second part was to set up the infrastructure for all disconnected devices. We knew that by the end of 2007, we would have millions of new Internet-capable devices out there. These devices did not exist in 2007 when we started, so we needed to start figuring out how we make all this stuff work together. What services do we develop? How do we develop the services in a very quick fashion? And that’s when we started looking at SOA, and what was available commercially and open source to design, build, manage, and govern a SOA.
CIO: What are you using on the embedded devices? How are you developing the embedded applications?
Ciurana: Some have a custom operating system that right now escapes me. The others are UNIX based. We’re a big UNIX shop up and down the stack, from the hand-held devices all the way to our back-end servers. We have some Solaris for some mission-critical stuff.
CIO: And what application development platform are they using to actually build the applications that they’re running on these embedded operating systems?
CIO: And then what are you using on the database side? Is it MySQL or some open-source database?
Ciurana: We’re using a very large commercial database.
CIO: Can you say which one it is?
Ciurana: Um, no. I cannot say directly who they are, but they have an office in Redwood City [laughs].
CIO: It sounds to me like the embedded device actually interfaces with the PC app, and then your PC app would be what actually interfaces with your Web applications … and I’m guessing you have some sort of Web services communications layer between the PC app and the back-end infrastructure?
Ciurana: That is correct. In broad strokes, that’s how it works. The actual Web services communicating between the layers, between the PC app and the back-end system; we have several variations of that. Most of it is SOAP. We have some calls which basically just help the server get or post something and that’s it. And then we have a number of Web services for other operations. We have a mechanism for uploading data from other devices. That one is just a straight form post with an attachment to a server, and then the server does something to store it.
CIO: The design and development process: How do you start this? How do you govern this? How are you moving this from concept to code?
Ciurana: I don’t think we’re like any other company that’s building consumer products out there. We have a BRD—a Business Requirement Document—that eventually becomes a Product Requirement Document. All the development teams participate into making decisions that figure out what needs to happen to execute on those plans, and then we get up and running. When we come to the actual execution, we have several different teams that are working on Web engineering.
We have my team, which consists of infrastructure. We are responsible for the overall system architecture, and that includes the connective product and the Leapfrog.com store. Then we have our data services organization. These guys are super smart at doing data modeling and data services and so on. And then the Web team does content management and all the core Web stuff, Ajax, GWT, etc. So those are the three major components.
Now, when it comes to the execution, because of the time constraints we had, we were looking at how do we do this quickly and cheaply—right?—because when you restructure, you don’t want to have a lot of money floating around. And we looked at, again, the commercial outfits that have helped us in consulting, software, and development. We found it was too expensive, and they didn’t have the skills we needed. So we decided to look at alternatives. We ended up choosing open-source technologies, and talking to the guys running the open source projects. We had them come in and work on a consulting capacity, and help us build the system.
We are actually getting much better hourly rates than we were getting with any of the big outfits that, you know, starts with an “A” or starts with an “I”—OK?—much better prices, and we’re also getting much, much better expertise. So, for example, if we want to do GWT, we hire one of the guys who is super active in the GWT community. When we came to databases and how do we structure all this stuff, we got one of the guys who authored one of the best-selling books on SQL to work with us.
At the end of the day, we have assembled one of the best teams in the world. We have some of the Mozilla Foundation guys working with us, helping us build this thing really, really fast. It’s actually cheaper than doing it through another outfit, and we’re getting a lot better service and a lot better turnaround time than we would have otherwise. I think that covers the first part of your question.
Ciurana: Then wasn’t the other part something about the architecture?
CIO: Yes. How is the architecture structured? You talked about development teams all over the world. Can you sketch out how the development organization is structured, what responsibilities are being dealt with where, how are you communicating on and agreeing on what aspects need to be exposed as interfaces, how those interfaces are going to operate, the governance of these interfaces, the development and governance of these interfaces?
Ciurana: Okay, so basically the architecture is set up in such a way that we have distinct layers. The first layer is the PC app that’s done by the PC app group; super smart, super sharp guys working in Windows and Mac and so on. Then we have the Web layer. That is locally done. The whole thing is done in our offices, but we have consultants from open-source projects working with us.
Then the next layer we call the backbone, which is a cluster of ESBs that are acting as a switchboard between all the communications coming from the outside world, whether it’s Web, PC apps, our partners, data consolidation, etc.; it’s all going through an ESB. And then we have our services layer that we call the “tailbone” or the “back-end,” which is another cluster of ESBs and servers that are providing the actual implementation for a lot of the services. The backbone provides switchboard capabilities, workflow, and transaction. And the tailbone does all the heavy lifting and implementations for each individual service.
In terms of how the development teams are organized, I would say about 40% of the development team has their offices in California. The rest are consultants that are based out of Colorado, Mexico, Canada, Belgium, Malta—we’ve got people working on this from all over the world. Our QA environment is also set up in such a way that we work on stuff here in California, and at an operation in Vietnam. Our monitoring is done in California. We also had an organization in Australia working with us on that. So we’re globalized in terms of how we are managing this environment.
CIO:How are you managing the teams? Where are you documenting your interfaces, your APIs, for want of a better term, how are you coordinating application development requirements? Give us a sense of the software tools and methodologies you’re using to keep everybody on track to get this thing done in 18 months.
Ciurana: Basically, we are following, again, an open-source model. Like I said, we have a Business Requirement Document and a Product Requirement Document, and from there we have a number of specs. Everything that needs to talk to the outside world has a spec that describes the protocol, data exchanges; things like that. That is shared among all the participants, and that is usually some kind of [Microsoft] Word document. Then, when we start coding and putting it all together, we have a number of development and QA environments set up, so we are always rolling out.
We have a very aggressive rollout schedule. Every few weeks we’re rolling out either something new to production or a patch. And that’s ongoing. So, for example, in the last two weeks one went into production. That was last Thursday. This Thursday we came in with a patch for some bug we found in production, and additional functionality. And we’re continuously rolling out. So the rule is, release early, release often, fix what you’ve got, and just try to work out all the protocol communications issues as soon as possible. That is one of the reasons why we settled on the SOA and Java technologies, because they enable us to let people work with whatever tools they want to use personally, but come into the environment with well-understood protocols and well-understood mechanisms.
So, in terms of the development team, we are very flexible. We believe that if you’re a developer and you’re a professional, you know better what makes you productive, so I don’t care how your stuff works, how you build it, how you make it, as long as it meets the spec, as long as it builds when we check it into the source library, and as long as it works with everything else. That helps us keep things moving quickly.
CIO: You talked about building using an open-source model. But how would you describe the development methodology? Would you describe it as an Agile development methodology? When you start with an 18-month window and the organization says, ‘This is what we need to get done in 18 months,’ in some ways the antithesis of Agile, which says we can only build this or that in 30 days. In a way, with Agile, what you’re able to build defines the timeline, not what you want to build. Feel free to disagree with me. And I’m not saying that Agile is right or wrong. I’m just asking the question, how would you describe your development methodology?
Ciurana: Basically we have Agile with a little special sauce. We do try to do Agile. We do peer programming. We do a lot of peer review. We argue a lot. Essentially, it’s Agile with a flavor of, “We do whatever needs to be done and we just get it done.”
Some corners of the software I am sure are not as well-structured and pretty as they would be if we had more time to build them. But on the other hand, we have a working system. We already have hundreds of thousands of devices in the last couple of months that are actually using our systems. And we have Leapfrog.com that has been in production since October .
So the methodology is, let’s build it, consult, figure out, work on the prototype, and release a lot. And that works really great. Part of what we have used for coordination is Internet relay chat [IRC], which is a chat protocol used by a lot of university kids. But it’s also often used by open-source projects. It’s like WebEx, but it’s on 24 hours a day, it’s free, and there are many free clients for it. So we [use IRC to] do a lot of the coordination and discussion, especially with the remote guys and the guys who are building the contracted parts of this.
I think our combination of these things is what’s helping us deliver quickly, and the methodology is a little bit of, “Let’s just get it done.” I wish I had a smarter answer than that. But basically, that’s what we do. That creates some friction, by the way, sometimes, with other parts of the organization which are more structured, more corporate; the IT guys and so on. That’s normal. They want more control, more governance stuff, etc., whereas we’re in the mode of, “Holy smokes! We’ve got to get this done!” So we’re going to do whatever we need to do.
CIO: Yes. You know, my job here is not to try to find examples of some perfected notion of how to develop service-oriented architecture, but rather, to bring to the readers of CIO, the listeners of these podcasts, insight into what’s really happening where the rubber meets the road. And the best part of this job, from my perspective, is I’m talking to a lot of folks who do what you do. And everybody is doing it differently. I think that’s one of the beauties of a service-oriented architecture, in that what’s behind those interfaces that are the touchstones, the connecting points between these application services and applications overall, is it doesn’t really matter. As long as the bolt is able to go into the nut and you can screw it down, things are going to work.
Ciurana: Absolutely. At the end of the day, if we look at what it costs us to build the stuff that we needed in 2007, and what it costs us to build what we are doing in 2008, we’ve spent about 30 percent of what we would have spent otherwise if we had gone with the more structured approach, or if we had brought one of the big consulting firms, or if we had gone with commercial products or their ESBs of the app servers and so on. We spent a lot less money, and we’re delivering a lot faster than we would have otherwise.
Look, the numbers don’t lie. We’re 30 percent of the cost, and most of that went to one big commercial product in this whole mix that’s about 20 percent of the total cost. Everything else has gone open source, and developed as quickly as possible. And it’s working.
CIO: So you’re doing “DIY SOA.” No commercial SOA stacks, emphasis on open-source, emphasis on Agile?
Ciurana:Well, for example, I’ve been using SOAP for a long time. And the more I used it, the more convinced I became that its main design was to help Microsoft and IBM sell consulting and tools. It’s way too complicated. There are better ways of doing things that don’t need to be that heavy for every application.
Now, the problem that I see sometimes is that people think SOA is SOAP or SOAP is SOA, and they push aside all the other things that have to go into SOA. So the approach we have taken at Leapfrog is best-of-breed. Let’s quickly figure out what the problem is, let’s understand what the business requirements are that we need to meet, and let’s find the best thing that is going to help us to fix this problem. That’s how we arrive at a means for having an effective mix of some commercial, some open-source, some home-grown, and some consulting. We just try to get the balance among those things.
So for a lot the things, like the connective apps and the PCs talking to the back-end systems and so on, we saw that SOAP made sense technically, and so we went with it. As we learned more about the system, we found there were other technologies for some of the requirements that were going to work better, so we started using those.
CIO:And what you’re really making is an argument that even experienced people in this industry confuse the technology with the architecture. I’ve heard people say that SOA is a technology and, in fact, you’re bringing out the point that SOA is not a technology, but rather, that you cobble together several technologies to accomplish a service orientation.
Ciurana: Exactly. It’s about how do you provide services that are aligned to the business as quickly as possible, and design patterns that allow you to mix and match legacy systems and new technologies. That’s really what SOA is about.
The vendors, of course, want to come in and sell you a stack. For example, I know a vendor that likes to come in and say, “Without this and that from us, your SOA just won’t scale.” What they’re trying to do is lock you in. When you start using their products in production and integrating and so on, if you ever need to go outside their black box, you start having a lot of problems and costs—because vendor stacks in general often work well with themselves, and not very well with the rest of the world, regardless of what the marketing material says. That’s one of the reasons why, when you are actually integrating all these types of systems, it makes sense to use things out in the [open-source] community that are intended to interconnect products from wherever or whoever they come from, over whatever protocol, and over whatever data mechanism that you happen to have, independent of a specific vendor’s flavor of things. And if any issues crop up, well, you have the source code and can immediately troubleshoot, without waiting for a vendor to look inside their black box and report back to you whatever they find.
CIO: And let’s talk just for a moment about your relationship with Mule. What drove you to bring Mule into your architectural mix, how are you using it, what benefits do you feel you’re getting, and what challenges have you encountered with it?
Ciurana: Okay. That’s a loaded question. Basically, I was working for this very large retailer. I was commissioned with other people to build also the next generation of e-commerce systems for those guys. At the time we were building our stuff, we identified that we needed some kind of ESB technology. We ran a number of proof of concepts with every vendor you can imagine, both open source and commercial. We set up a couple of SOAP servers for them and said, “Okay, here’s everything. Here’s what we need. Here’s the timeframe that you have. Let’s see what we get.”
During that evaluation, we learned a lot. We ended up liking the Mule product, and we really liked one of the commercial guys. The difference is the commercial guy, just for the basic licenses, was going to cost something like $350,000. We said, ‘Forget it. Too expensive.’
We decided Mule was the way to go, and we ran with it. Eventually I wound up [leaving the retailer and] going to Leapfrog. And when I got here, I decided to do another SOA. Mule looked good, and by then they had a real company behind it. They had the ability to provide support and training. So we brought them in, first on a limited basis, with a couple of servers. We got what we expected, or better, so we decided to go for the enterprise license and make a big commitment. In general, working with the Mule software has been both exhilarating and frustrating. The implementation has always been its weaker link.
CIO: So explain to the readers the role that Mule serves in the architecture, and then give us an example of an implementation problem that you encountered that you needed to get resolved.
Ciurana: Okay. Well, basically, Mule connects everything—our back-end systems, databases, partners; all of our connective product, absolutely everything that’s happening in the Web environment, is going through Mule. Everything. If two boxes, two systems, that talk to each other in any shape or fashion, that’s going through Mule. So if something breaks, that means that my whole environment dies. I mean, that’s mission-critical, right?
So one of the things that we had a problem with—this is going to get a little technical—in one of the protocols that we were passing through, the front end and the back end, we found that some of the bits were not working. I mean, this is pretty obscure. I got on the phone with MuleSource at roughly 4:00 in the afternoon, and I said, “We just found the problem. We need to fix it. I need to have it fixed by 8:00 in the morning tomorrow because this needs to go to staging. We’re pushing it out the day after tomorrow. So I need it now.” And, sure enough, within an hour, I had one of their consultants working with us. By 9:00 we had found the issue was caused by a combination of something they did in the product and something we did in our code. By 11:30 at night we had fixed the problem. By 8:00 in the morning we were able to push this staging. Everyone is happy.
CIO: Great. I really appreciate the time you spent talking to us today, Eugene. It’s been really interesting
Ciurana: You, too, sir. Bye-bye.
CIO: I’ll check back with you at the end of the 18 months.
Ciurana: [laugh] Well, the end of the 18 months is October 3rd, so . . . you’ll actually be able to see the finished product.
CIO: Fantastic. I’ll look for the press release on the wire, and then maybe I’ll have to pick up a Leapfrog toy for my kids, and we’ll check the whole thing out.