Agile Comes to Data Integration
Informatica is aiming to ease the pain of data integration with a new platform designed to allow businesses to rapidly prototype and validate before sending projects to development.
Wed, February 19, 2014
CIO — When it comes to big data, it's easy to get excited about things like predictive analytics and prescriptive analytics that hold up the promise of radical change in the way organizations do business. But the guts of any production big data project—even the most bleeding-edge prescriptive analytics—is data integration.
"Data integration projects take too long," says Todd Goldman, vice president and general manager of Enterprise Data Integration & Data Quality at data integration specialist Informatica. "Part of the reason they take too long is because data integration projects start with a specification from the business, usually in the form of an Excel spreadsheet."
Manual Data Integration Is Like a Game of Telephone
That spreadsheet gets passed to developers who then do a great deal of work on the project before any validation gets done, Goldman says. It's like the child's game of Telephone. The spec gets passed on to someone who does a little work before it gets passed on to someone else who does more work and so on. Only at the very end do you check the output. It's a manual process rife with inefficiencies.
"Every time you request information to make a decision, your employees manually go gather data and roll it up," says Jared Hillam, EIM Practice Director with Intricity, an IT services company that specializes in data management.
"So, every time a request for data is made, you are reworking the data gathering and roll-up logic. The unfortunate part is that the process is error prone and slow—with the business complaining that it does not get the data it asked for," Hillam says. "Having specialized in the implementation of data warehouses, we know first-hand that there is a better way. What is needed is an agile data integration platform that involves the business user throughout the process, from requirements gathering, to requirements validation, to rapidly prototyping the solution, to analysis and profiling and then to testing. IT then can do the final deployment and on-going monitoring of the environment. With the business and IT now happy, it's a win-win for all."
Agile Meets Data Integration
In a nutshell, that's the idea behind the new Informatica 9.6 platform, which steals a page from agile development processes and brings it to data integration, allowing organizations to accelerate data integration projects by a factor of five compared with traditional processes, according to Goldman.
"Instead of using Excel and waiting to validate after the developer does a whole bunch of work, we're going to give the business the ability to do their own validation before it goes to the developer," Goldman says. "They can turn their specification into a real, working prototype. Then the developer adds scalability, reliability and performance."