When Your CRM System Passes 1 Million Records
Few enterprise software systems have to scale to millions of records. Even in a smallish enterprise, though, a CRM system can rapidly grow beyond that level, particularly if it's connected to e-mail blasters and marketing automation systems. What do you have to look for and do differently in these bigger CRM environments?
Tue, March 02, 2010
CIO —
There isn't a sales force in the world that says it has enough Leads. And you won't find many marketing VPs who want to do fewer campaigns. So there's a never-ending stream of new leads, prospect interactions, and conversations to be stored in the CRM system. At companies in consumer markets, open source software, and other categories it's not unusual to find a million leads or more. But that's just the beginning: if you're using the latest marketing automation system, every e-mail, web download, and prospect response is recorded in the CRM system. And if you have a large call center, every call and e-mail exchange should be recorded well.
[ For timely data center news and expert advice on data center strategy, see CIO.com's Data Center Drilldown section. ]
This can mean millions of records, with thousands of new ones every day. No other system in most enterprises needs to deal with this kind of data flow, particularly with the number of simultaneous users that a CRM may have. If you're growing the scope of your current system, or consolidating several division-level CRM systems, here are things to consider as the data scales upward.
Platform Performance
Nearly any real CRM system will have reasonable performance for most use cases even with data sets of this size. But there are some features that will inevitably start to slow down: reports, dashboards, or any functionality that involves scanning all records. While some of the problem can be solved with the usual workarounds (de-normalizing data, creating analytic digests during off-hours, pre-joining views, etc.), anything that involves an ad-hoc select or upsert can't fundamentally be helped without throwing hardware or coders at the problem.
User Interface Performance
Under most conditions, the responsiveness of the CRM user interface will be fine with large data sets because users are typically working with only one record at a time. But if your CRM system runs inside a browser, watch out for UI features that loading of large lists for the user. Some systems may try to load thousands of names into a scrolling list, with appalling performance consequences (particularly on Firefox).
Third Party Apps
While the core CRM platform will probably handle really big data sets, we've seen cases where supposedly enterprise-scalable third party apps couldn't even load the data, let alone process it properly. The combinatorial explosion that occurs within some applications' algorithms can lead to corner cases that are tough to anticipate. As always, run pilots with the complete data set before you put third-party apps into production.
