CIO — HANG IN THERE. That was the first plea American Eagle Outfitters (AEO) offered its nearly 1.5 million Facebook friends three days after its trio of e-commerce sites had crashed.

The online retail outlets—ae.com, aerie.com and 77kids.com, which target teens and kids—suffered what American Eagle Outfitters (AEO) termed a "major incident" on July 19. "This has caused all of our sites to go down," a Facebook update said on July 22, at 1:26 p.m. "We're working together 24/7 to get them up and running as soon as possible. Hang in there....we'll make it up to you. Check back soon."

Those back-to-school shoppers who checked back too soon at the sites received this message: "We're making updates to our sites." This was no planned maintenance, however. Behind the scenes, AEO and its external hosting provider IBM (IBM) were scrambling for several days as the sites stayed dark.

The reasons for the four-day outage (and subsequent four more days of technical aftershocks) related to backup and disaster-recovery technologies and processes that failed, according to several articles on retail tech site StorefrontBacktalk.com.

As the articles describe, atypical and concurrent failures with IBM's hosting servers and backup plans as well as with Oracle's (ORCL) Data Guard utility program ultimately proved to be the sources of problems. (Stephanie Oschwald, a spokesperson for Pittsburgh-based AEO, did not return several messages seeking comment. IBM and Oracle declined to comment for this article.)

[ Read about e-tailer Zappos.com's $1.6 million pricing mistake ]

On July 23, at 5:39 p.m., AEO announced on Facebook that the sites were A-OK. "We had a few glitches but we're back! We missed you while we were gone. Free shipping now thru Monday, July 26 11:59 PST. Automatic on your entire purchase (US and Canada only)."

However, noted StorefrontBacktalk.com, four more days of technological healing lie in wait for the AEO site, "when it limped along with various parts not functioning until Tuesday afternoon, July 27th."

An Unprecedented Outage

The shock of a 100-hour e-commerce outage now over and the damage done to their critical back-to-school sales period, executives of the $3 billion AEO have to be left scratching their heads in frustration—and wondering just how much "teeth" they had written into their service-level agreements (SLAs) with IBM. (For the record, StorefrontBacktalk.com reported that AEO had recently filled the position of "Manager—Business Continuity & Disaster Recovery," and not a moment too soon.)

AEO's websites weren't the first to go down unplanned, nor will they be the last. Dating back to the mid-1990s, history is littered with tales of site outages, complex IT fixes and lost revenue—embarrassing for the companies and career killers for those in charge. Any Amazon.com unplanned downtime, for instance, is still reported with as much zeal as a Jennifer Aniston "new boyfriend" sighting.

Continue Reading