“If you can’t tell it was this production line on this shift on these days,” Gonzalez says, “then you’re stuck with an estimate and doing an expansive recall just to be safe.”

Manufacturing execution systems, such as packages from Atos Origin or Manugistics, keep that kind of data. SmartOps offers a supply chain dashboard that managers inside and outside the factory can use to watch manufacturing activity and be alerted to problems, such as failures in quality control. (For more on manufacturing execution systems, see "Software for Hard Times.")

On the deodorizer recall, Gonzalez’s company could trace the offending ingredient back to a supplier in the U.K. But that supplier didn’t keep records to show whether the raw materials had come in contact with other ingredients. “All we could do was assume that everything from the U.K. supplier was contaminated,” he says, adding that the recall cost “several million dollars.”

Sometimes data isn’t stored long enough or outside companies hired to manage your warehouses or ship your products don’t retain their data as long as you do. Barkai, at Manufacturing Insights, remembers wanting to study one year of supply chain activities at a U.S. auto maker. “I was told I would not be able to receive [the data] because they had limited server space, so they purged it,” he says. “Someone decided six months was good enough.”

When deciding how long to hold data, factor in the product’s life span, as well as any local health department requirements, federal antiterrorism legislation and the potential for litigation if something goes wrong, recommends a data security manager at ConAgra, who asked not to be named. He said data collection during a recall is “an arduous task” at the company, but “not a scramble.” (For more tips, check out "5 Recall Best Practices.")

How to Manage a Recall

As Blasgen explains, the central goal in a recall is to figure out how much of the bad product was manufactured and where it went.

To do that, he recommends starting by querying your manufacturing execution system, which keeps data on production lines, product batches, ingredients and conditions inside the factory when products were made. ConAgra, according to its website and other corporate communications, uses a mix of Manugistics and SAP software in its factories to schedule and monitor manufacturing runs.

Next, Blasgen says, IT managers should pull shipping records from their transportation management systems. (Kewill Ship and HighJump Software are typical systems.) Search for date, lot or batch codes to figure out which distribution centers or customers were sent the suspected bad product, Blasgen says. Then match that information against data in the distribution centers’ and customers’ receiving systems.

Those are the steps Blasgen recommends. But most supply chains are not configured to allow a CIO to follow them. “Do you see what’s happening?” Blasgen asks. “Multiple systems with their own ways of querying a nd no magic ‘recall button’ to press for any of this.”

Fixing a Hole Where the Rain Leaked In

ConAgra had streamlined its supply chain with its $300 million “Project Nucleus” started in 2003 and mostly led by Blasgen. The company closed factories, consolidated distribution centers and standardized on SAP enterprise resource planning software. Blasgen says that any company that reworks its supply chain to be more efficient will make and move products faster. That’s good when the products are good and bad when they’re not.

Steve David, the former Procter & Gamble CIO, notes the irony. “You take the bad with the good,” he says.

In recent financial statements filed with the Securities and Exchange Commission, ConAgra claims savings of $275 million so far from its supply chain revamp. That pot has helped offset the $78 million ConAgra reportedly has spent to date on the recall—costs that include customer notifications, installing and staffing a toll-free hotline, consumer refunds, and collecting and disposing of bad peanut butter.

Another $15 million to $20 million went to overhauling the Sylvester factory. ConAgra bought a new peanut roaster, upgraded finished-product testing procedures, built new factory walls and designed other ways to keep peanut dust and raw peanuts away from already roasted peanuts (heating to 165 degrees kills salmonella). It also put up a new roof and installed a new sprinkler system. Why?

Because the company has a theory as to how salmonella got into Peter Pan in the first place. For an undetermined period of time, broken sprinkler heads dripped inside the Sylvester plant while rainwater seeped through the factory’s leaky roof.

The moisture from those sources may have awakened dormant salmonella bacteria in peanut dust or in stocks of raw peanuts. That what ConAgra’s Colo told Congress in April. ConAgra’s finished-product tests missed the contamination, Colo testified. The company then trucked jars to distribution centers and from there to Wal-Mart, Dollar General, Kroger and other supermarkets, warehouse clubs, food distributors and restaurants across the U.S.

At last count, the Centers for Disease Control and Prevention reports that 628 people in 47 states got sick.