The addition of a new software routine caused the multi-hour failure of Research in Motion's BlackBerry messaging service in North America earlier this week, the company said Friday.
In its first statement explaining the outage, which began on Tuesday evening and continued until Wednesday morning, the company said the introduction of a new software routine that was supposed to optimize system cache memory led to the problems.
What did you do when BlackBerry service went down?
"The system routine was expected to be non-impacting with respect to the real-time operation of the BlackBerry infrastructure, but the pre-testing of the system routine proved to be insufficient," Research in Motion (RIM) said in a statement.
Things were made worse by the poor performance of the company's backup system.
"Although the backup system and failover process had been repeatedly and successfully tested previously, the failover process did not fully perform to RIM's expectations in this situation and therefore caused further delay in restoring service and processing the resulting message queue," it said.
In pointing to the software fault, the company also said it had ruled out any security or capacity issues or the failure of hardware or core software.
RIM, which apologized to customers for the outage, said its examination of the problem was not yet complete and that more information would be provided as it becomes available.
"RIM has already identified certain aspects of its testing, monitoring and recovery processes that will be enhanced as a result of the incident and in order to prevent recurrence," it said.