by Thomas Wailgum

The Man Behind MasterCard’s 100 Terabyte Data Warehouse

Jul 17, 200812 mins
IT LeadershipNetworking

MasterCard's head of global technology, Rob Reeg, talks about his new job, MasterCard's massive data warehouse, just how fast credit-card transactions really fly and his personal experience with credit-card fraud.

Rob Reeg took over as president of MasterCard‘s Global Technology and Operations in May 2008, the de facto chief of IT at the $4 billion credit-card and electronic payment provider.


How to Be a Supremely Productive Person: A Chat with CIO John Halamka

IT-Led Innovation at MasterCard

How to Thrive as an Extreme CIO

This is a glimpse into his new world: On behalf of 25,000 financial institutions, MasterCard’s worldwide network in 2007 processed 18.7 billion transactions totaling some $2.3 trillion. On an hourly basis, the network has to manage 140 million transactions, with a response time of 140 milliseconds per transaction. And 99.999 percent network availability is standard operating procedure.

All of this wasn’t totally new to Reeg. He had served in the position on an interim basis since January 2008. And before that he was MasterCard’s CTO. “So it was a fairly easy transition,” he says.

Though the transition may have been a piece of cake, his job’s demands are anything but. Ridiculous network requirements aside, Reeg is responsible for managing MasterCard’s global IT environment, headquartered in St. Louis and with operations in 210 countries, preventing frauds and credit-card scams, innovating with transaction-processing systems and enhancing financial institutions’ product offerings gleaned from MasterCard’s data warehouse. (To read more about MasterCard’s IT innovation and what Reeg’s predecessor did, see “IT-Led Innovation at MasterCard.”) Senior Editor Thomas Wailgum talked to Reeg about his biggest worries, how his credit-card data got compromised, and just how “Priceless” his IT staff is.

CIO: What are your new responsibilities?

Rob Reeg: There are five key functions I’m focused on: 1. Our global network and how those 18.7 billion transactions flow. 2. Making sure we leverage that network to provide very fast, reliable payment processing. 3. On top of that platform, how do we provide our customers with custom and differentiated payment solutions. 4. How do we provide support to customers as they use our products and services. 5. How do we leverage that gold mine of data that occurs when you have 18.7 billion transactions that you’re processing.

Rob Reeg
MasterCard’s Rob Reeg

Who are MasterCard’s customers?

Reeg: Our customers are typically the banks. We’re in what we call a four party model. There are card holders, who have a relationship with an issuing bank; so that’s two parties in the model. On other side there are the merchants have a relationship with their bank, an acquiring bank. Those are the four primary stakeholders in any kind of payment transaction.

We sit in the middle, between the issuing and acquiring banks, making sure those payments occur as they should and then move the money appropriately.

How big of an infrastructure do you have to support and maintain? It must be huge.

Reeg: Actually from a pure server footprint standpoint, in the data center, we probably have fewer actual footprint servers because of techniques like virtualization that help us leverage one box to do multiple things.

Where it gets interesting is philosophically: We try to put [transaction] processing as close to our customers, the banks, as possible. When we talk about the global network, we have small servers that sit with the bank customers that connect to our network. What it does is it gives us intelligence there at the end of the network. So as a transaction comes through, we can take a look at that transaction and decide how do we best process that transaction for the benefit of all those four parties in the model.

As to processing, the majority of transactions we’re looking at relate to how do we process them as fast as possible in the most accurate way. The way to do that is by peer to peer: If you’re using your card in Europe, in London, say, and you swipe your card as you check out of hotel, we can route that transaction to the hotel’s acquiring bank in London directly to your issuing bank and get that message back for approval without ever going through St. Louis or some big data center in the middle of all that.

You talk about competitive differentiators like that, but that message isn’t going to consumers, who use the cards. So does that networking innovation message go to the banks?

Reeg: We obviously talk to banks and then we’ll do pilots [with them]. For example, we just introduced our card we call PayPass that leverages contactless technology. We did a pilot with the New York Transit System and the bank that supports the transit system. So people take their PayPass card, touch it against the entry device or turnstile, and get into the subway, passing through very quickly.

So it’s a combination: We’ll work with banks and merchants, and with banks on programs they want to introduce to cardholders. Leveraging the data warehouse, we do help banks do targeted marketing and targeting rewards programs, for example, if a bank wants to do a special deal with a big electronics merchant to incent customers to use their card there. We may help them segment a portion of their portfolio that we believe, based on the data, would be interested in that kind of service.

When a consumer uses a MasterCard in a store, the transaction is usually seamless and fast. So what networking things happen in those seconds that I’m standing there at the counter?

Reeg: It’s pretty interesting, because like you said, the vast majority of all payments you never really think about. It always works. It’s like turning on the tap and expecting water to come out.

But what really happens is: You’re at the hotel, and they have a card reader on their desk when you check out. That card reader is swiped or in the case of PayPass [is used as] contactless payment, and it transmits information off your card to the merchant. The merchant is then connected to their acquiring bank, and their acquiring bank takes that transaction, knows it’s a MasterCard transaction and gives it to us. We take the transaction, look at how do we best route this transaction to get it approved and send the routing information off.

The routing between the acquiring bank and the issuing bank, which is what MasterCard does, we typically call that “switching the transaction.” That happens in a 140 milliseconds. An eye blink takes about 400 milliseconds. So about three transactions go through every time you blink your eyes.

At the other end of the network, as we switch to the issuer, they take it and look at your account. They’ll look at things like how much room do you still have on your card. If it’s a debit card: Is your debit account OK to take this [charge]. We have fraud tools that get involved to look at it to try to make sure it is not a fraudulent transaction.

And then they pass back the approval to us, and we give it back to acquirer, who gives it back to the merchant. All that happens in just a few seconds — just a tremendous amount of processing.

How big will contactless payment become?

Reeg: It’s actually rolled out now. You’ll see terminals at different places and devices, like in movie theaters, and fast-food restaurants that accept PayPass.

How have you prepared for this?

Reeg: One of the things that MasterCard did really well a few years ago is that we rewrote our core processing systems. We had some really good technology architects who did a great job thinking about what we need to have to support the future. They came up with this concept: Our systems really needed to be agnostic as far as it went to the point of interaction.

So whether a transaction is originated via a contactless card, a cell phone, the Internet or just a regular swipe, it shouldn’t matter as long as we build the system that is open and can accept the transaction coming from any device. We invested about $160 million in that rewrite, and it has really paid off well, now that we see more and more different kinds of devices that start a transaction.

How large is MasterCard’s data warehouse?

Reeg: The ultimate goal for MasterCard IT is how do we leverage technology to drive business. The tenet is: We should look to see how we can make every transaction more valuable to all those four stakeholders in our four-party model. The data warehouse is a key component of that whole strategy. Look at the warehouse itself: There’s obviously a wealth of data just from the sheer volume of transactions that are being processed, and we know that we can take that data and help our customers better market to their own card holders or better serve their own merchants.

There’s over 100 Terabytes of transaction data in our warehouse now. We really believe there’s a capability that we’re going to have 1.8 Petabytes as this continues to mushroom.

Fraud prevention is a hot topic these days. What’s your role in that?

Reeg: We build specific tools for our banks to use to help augment their own fraud-prevention services. It’s funny, the last time I was in Europe my personal card data was compromised. My bank called me before I knew anything about it and told me they were seeing unusual activity. Again, when you can leverage the data of a 100 Terabyte data warehouse, and build neural network type models based on spending patterns and what merchants I typically go to and what places I usually visit and all that, then there’s s wealth of data that our customers can use and we can give them tools to help use to help prevent fraud.

It’s a constant, constant battle, and something we have to be very vigilant about because the bad guys out there are just as determined to use things the wrong way.

How can data warehouses and large infrastructure operations like yours be green?

Reeg: We actually believe there’s a financial benefit from that as well just in terms of energy consumption. We custom build our data centers, and we had some opportunities to put in, for example, extra high ceilings to disperse some heat. We’re heavily into virtualization so that we can reduce the numbers of footprints.

We operate in 210 countries around the world, with 25,000 different financial institutions. In doing that we leverage a global workforce. So part of that is: if I have programmers in Asia, they’re using the same server that my programmers in U.S. are using, but at different times of the day. So I can get better usage of a server by having a global workforce as opposed to having to buy multiple servers, if everybody’s in the same location.

How critical is IT to MasterCard’s success?

Reeg: This has to be one of the best IT jobs in the world. A lot of companies struggle with: Are the IT goals aligned with the business goals? There’s no difference here. Our IT goals are the business goals and unless we’re absolutely linked, MasterCard doesn’t function as effectively as it could.

Do you feel extra pressure because of that?

Reeg: I think it’s just the business we are in. The difference between the president’s role and the CTO role [I previously had] is that you have to be thinking from a business standpoint all the time. How does any change we make help make the business be better for our customers, cardholders and merchants. How do we leverage technology to be better for all the stakeholders? It really drives the business.

For example, we have a group of people totally focused on the best kind of payment products [for banks], and they work with the IT guys to say: How do we best build that in our infrastructure? Then you have a team that’s focused on working directly with the banks and saying: Here’s the best MasterCard product for you. We may make tweaks to the IT infrastructure to make that the best possible service for the bank, but it really comes down to, for our group, our most important contribution to MasterCard is the quality, speed, strategy and innovation for those IT services we provide. And doing it at the very best possible cost.

What keeps you up at night?

Reeg: I’ve given up sleeping. [Laughs] We don’t have concerns, really, but we have some interesting opportunities we’re focused on. We just announced a new processing platform called the Integrated Processing Solution, and the first customer went live yesterday. That one has been a big part of focus for last few months as we’ve been building this platform and bringing this customer up on it.

So is being innovative IT and enabling innovation throughout MasterCard a top priority for you?

Reeg: Absolutely. It’s part of the fun things about this job. This whole industry changes on a very fast basis. When you look at payments overall, the biggest opportunity we have is not against another payment process. It’s with cash and check, because the majority of all payment transactions are still done via cash and check.

That’s surprising. I travel with 10 bucks on me and my credit cards.

Reeg: You see it declining at a very fast rate every year. But today over half of the transactions are cash and check.

How do you keep the staff motivated on keeping the network up 99.999 percent of the time?

Reeg: There’s a phenomenon in this business: front of the wallet and back of the wallet. Most people in U.S. carry more than one payment card. And if one payment card doesn’t work, what do you typically do? You pull out the next one, right? And now that other card went to the front of your wallet. So one of the operating tenets that we have is that we never want to have a network issue that causes someone to move their card from front to back of wallet.

Does everyone at MasterCard walk around doing the “Priceless” bit? Or are you all sick of it?

Reeg: No! We love that commercial! Actually I just had another one. When we introduced the new processing platform, we had a little takeoff on the “priceless” commercial again. It came down to, and I really believe this, that the technology people at MasterCard are priceless. You spend so much time at work, you have got to feel good about coming in in the morning. If not, you’re doing the wrong thing.