Salesforce.com Nailed for Downtime: But Your In-House CRM and ERP Record May Be Worse

On January 6, Salesforce.com, the software-as-a-service CRM vendor, suffered a “service disruption” at 12:39 p.m. Pacific time. The technical reason for the outage: a core network device had failed, due to memory allocation errors. The backup plan, which was supposed to trigger a cut-over to a redundant system, also failed.

“This resulted in a full service failure for all instances,” noted a post-incident explanation on the Trust.salesforce.com site, which provides real-time information on Salesforce.com system performance. (Ironically, it also failed when the SaaS CRM platform went down.)

Some 30 to 40 minutes later, after Salesforce.com techs initiated manual recovery steps, users found the SaaS site working again at full tilt. The dark cloud had lifted.

During the outage time, many customers relied on Twitter to communicate with each other about the site’s shutdown, seeing as how Trust.salesforce.com was unavailable.

The blogosphere erupted with what felt like a dose of software schandenfraude. “Salesforce Demonstrates How (Un)reliable SaaS Really Is,” noted one blogger. Chimed another blogger: Salesforce.com Crashes, with just a little too much glee.

Five days after the outage, technology consultant and blogger Jeffrey Kaplan wrote a more measured review of the incident, raising this question: “Salesforce.com’s Outage: Will It Derail the SaaS Market?”

“Salesforce.com’s outage re-ignites the debate about the reliability of web-based services, and will intensify the concerns of those IT and business decision-makers who have been reluctant to adopt on-demand solutions,” Kaplan wrote. “It also validates the claims of legacy software vendors that SaaS and cloud computing are not viable platforms for enterprise applications.”

Kaplan was correct in raising the point, given Salesforce’s track record. In late 2005 and early 2006, Salesforce.com customers endured a series of frustrating outages, one lasting nearly six hours. So you can bet that Oracle, SAP and other on-premise enterprise software vendors would seize on Salesforce.com’s latest misfortune. The on-premise software vendors are scared to death of SaaS.

Despite lingering (yet fading) concerns over integration and security, SaaS and on-demand software options are becoming mainstream. Salesforce.com now has nearly 52,000 customers and posted $749 million in total revenue for fiscal year 2008.

But here’s the maybe-not-so-obvious rub in all of this: Salesforce.com’s overall uptime is, more than likely, better than the uptime of your in-house CRM or ERP applications. That’s especially true if you have built your own: A June 2007 survey of 200 U.S. IT managers and senior leaders found that homegrown applications were the “prime culprits” of system downtime. Homegrown and custom-built applications represented up to 90 percent of some organizations application mix, noted a NetworkWorld article, and more than 80 percent of survey respondents relying more on homegrown applications than off-the-shelf packaged applications blamed software as the main cause of most outages.

A recent IDG News Service article observed that “Salesforce.com has had fewer significant outages than it did years ago when it first launched.” Kaplan, in his blog, added that “Salesforce.com’s uptime record is still the envy of many IT and business decision-makers.”

So just what is Salesforce.com’s uptime? I couldn’t track down a standard, across-the-board number (like, 99.99% uptime guaranteed) on either Salesforce.com or Trust.salesforce.com, which boasts of “high levels of availability across all Salesforce instances since inception.” (A media relations staffer couldn’t provide an immediate number.)

However, Salesforce.com CEO Marc Benioff has said in the past that system uptime has never fallen below 99 percent. (According to Salesforce.com’s most recent annual report, the company “has entered into service level agreements with a small number of its customers warranting certain levels of uptime reliability and performance and permitting those customers to receive credits or terminate their agreements in the event that the company fails to meet those levels.”)

One other point to consider: Vinnie Mirchandani, a former Gartner consultant and IT industry veteran whose Deal Architect consultancy counsels CIOs during negotiations with enterprise software vendors, writes in a February 2008 post that the 99.99 percent uptime requirement at service providers such as Salesforce.com is a nothing but a corporate myth. “I find the general assumption among us bloggers is [that] these providers have to be up 99.9999% of the time,” Mirchandani writes. “[It’s a] major crisis when any of them has a crisis. Because supposedly that is what corporate America has always got and continues to get…. Other than some sensitive and global apps, there is plenty of downtime in corporate data centers and those of big outsourcers like IBM and EDS.”

Remember that 99.9 percent uptime means that the application or service is down roughly nine hours each year, and 99.99 percent uptime translates to only 50 minutes of downtime a year. For the record, 99.999 percent is just 5 minutes per year.

So, how many 9s are you getting with your CRM app?