Taming the Transfer of Monster-Sized Files

Despite being insecure and outdated, the file transfer protocol (FTP) continues to be a popular method for sending large files between data centers, corporate divisions, branch offices or business partners. But watch out.

A study commissioned by Sterling Commerce, a subsidiary of AT&T that focuses on business and collaboration software, found that nearly a third of companies do the majority—as much as 80 percent—of their large file transfers using FTP. Yet, the same study found that 93 percent of companies experienced failures in FTP transmission. Another survey of manufacturers conducted by Sterling found that about 8 percent of their file transfers fail.

“Companies can no longer rely on the solutions to manage batch file transmission that they had put in place for mainframes,” says William McKinney, head of product marketing for the company’s managed file transfer (MFT) group. While FTP is free, it’s not appropriate for companies transferring medical data or retail-sales data to their data centers, he says. “The problem is that it was designed for interoperability, it wasn’t designed for security or automation. If something goes wrong, there’s no record of it—people just try it again.”

[ For timely data center news and expert advice on data center strategy, see CIO.com’s Data Center Drilldown section. ]

Large files moving back and forth between corporate groups and their data centers account for the greatest volume of transfers: Financial firms transfer checking account transactions or credit-card data; healthcare companies have to collate and synchronize data on specific procedures with the medical files; and retailers have to upload their latest sales data from their numerous stores.

Dumping FTP Has Compliance Benefits

Moving away from ad-hoc FTP data exchange can save companies money and headaches, not only during day-to-day operations but when being audited for compliance. For companies that want to—or for compliance reasons, have to—have assurance that large data files are transferred securely and reliably, adopting a managed file transfer solution makes sense, says Maureen Fleming, program director for IDC’s business process automation and deployment group.

“If you have a system that is not all that reliable, and it has an unacceptably high exception rate—and anything less than 99 percent is unacceptable—people will be wondering what happened to that file,” Fleming says. “If you cannot figure out what your exception rate is, then you are in trouble because it becomes a brand issue.”

The demand has helped the managed file transfer market grow to more than $500 million in 2009, according to Fleming. Vendors playing in this space include Axway (which bought Tumbleweed,) Ipswitch, IBM, Accellion and CA.

“There are a huge number of batch-oriented processes that are going on,” Fleming says. “CIOs have to ask, how often do we update and how often do we synchronize the data we have in the cloud with what we have in our data center.”

Finding transfer failures becomes very important to operating smoothly, says Sterling Commerce’s McKinney.

“Most people don’t realize the value of ‘managed’ in managed file transfer,” he says. “If you have a batch process that kicks off every night to the data center, when would you like to know that you had a problem with a transfer from one of your stores? At 12 o’clock when you start trying to use that data? No. If this transfer hasn’t started by 11:05, you want to get someone in there to fix it.”

Managed file transfer solutions offer not only assurances that transfers complete, but more security for the data in transit and better auditing capabilities to satisfy regulations, such as the Health Information Protection and Privacy Act (HIPAA) and Graham-Leach-Bliley (GLB).

Collaboration and Partner Pluses

In addition, companies looking at increased collaboration with partners will want assurances that only authorized people can view certain data and services, says Sam Morris, product marketing manager for Attachmate.

“In the majority of conversations we are having with customers, they are looking for ways to enable the immediate exchange of information between themselves and their partners and customers,” Morris says.

Companies that find themselves increasingly dealing with global partners, moving away from physical media and worried about security breaches could benefit from looking at managed file transfer solutions, Morris says.

“Just the ability to know that a file transfer has completed and completed with integrity is something that most companies want,” he says.

Follow everything from CIO.com on Twitter @CIOonline.