Snapdeal Revamps its Database to Augment Speed and Scalability

Summary:

As Snapdeal’s business grew 200 times, it needed its IT systems to scale and support its rapid growth. How its CIO made that happen.

Shopping is an endless thrill. It gives a high that’s unparalleled. It’s powerful enough to make the most apathetic shopper succumb to his latent shopaholic impulses. But in today’s fast paced life scouring shops and standing in painfully long queues makes shopping a gruelling experience. Online shopping has changed that. Today, buyers can just check out the multiple choices available on their laptops, snap up the best deals and their shopping is done. And one company that does that in style is Snapdeal. Snapdeal is one of India’s largest e-commerce companies.

The Organization:Launched in 2010, the online retailer boasts of a network of over 22 million members, over 4 million products across 500 categories and more than 20,000 sellers. The company caters to over 5,000 cities in India. Today, Snapdeal.com’s platform enables sellers to list products for sale on the site, manage inventory, and make up-to-the-moment pricing changes. While that sounds impressive, the amount of pressure this enormity of business puts on Snapdeal.com is overwhelming.

Its inventory and pricing management system had to process more than 500 writes per second (write operation in the database allows sellers to make changes to the price and inventory). In the meantime, buyers review and rate sellers for customer satisfaction based on their product experience, including shipping, delivery, and returns. All this had to happen in real time. However, as the company grew at a blistering pace, it was faced with the challenge of managing data real-time. “In two years, we have scaled more than 200 times—the number of products listed, the number of sellers, the amount of business they do, the number of servers, storage and the technology team—everything has grown 200 times,” states Amitabh Misra, VP- Engineering, Snapdeal.

Flipkart Drives Innovation Through Intelligent use of IT

The Business Case: To support its inventory and pricing system, Snapdeal had deployed 10 MongoDB NoSQL database servers with 5GB of data in DRAM as a cache in front of MySQL. As the seller base swelled and it started listing more products and making more changes in response to market dynamics, Snapdeal had to moderate the rate at which it was pushing updates—in terms of price and inventory. It had to spread them out evenly throughout the day as opposed to being able to absorb them in near real-time. Sometimes the updates had to wait for three hours and the sellers could not sell anything during that time. MongoDB’s response times also shot up from 5 milliseconds to more than a full second. This compromised the buyer’s shopping experience. Moreover, Snapdeal anticipated its business volume would rise around Diwali and Misra wanted to address the challenge before it impacted business. “We needed our system response times to complete within 10 milliseconds, 95-99 percent of the time—even while our platform processed up to 10,000 reads per second,” recalls Misra. With 500 percent growth in 2013, Snapdeal realized that it would require a high-throughput and low latency system that could scale from two million to 150 million records, from 10,000 to 30,000 reads per second, and from 30 to 500 writes per second. Misra started out by evaluating the available solutions on four parameters: Cost-effectiveness, the ability to perform concurrent reads and writes, scalability and performance.

Most of the systems he reviewed failed to meet at least one of the four criteria. But soon he was able to zero-in on an in-memory NoSQL database that met his criteria. First, it performed with predictable low latency with 95-99 percent of transactions within 10 milliseconds. Second, it had the highest throughput. Third, it delivered the highest price/performance and, on top of that, it offered the lowest cost solution in terms of both hardware requirements and ease of operations. “None of the other systems that we evaluated were fully viable options for us. It worked magically out of the box,” says Misra.

The Benefits: Today, Snapdeal has the ability to scale with the company’s growing business volume. “Now, we can push through huge price changes while maintaining the same response time experience on the buyer’s side—even with millions of buyers. That’s the biggest advantage,” says Misra. Today, Snapdeal knows that even if its business grows 10 folds its database can scale up to that level without compromising customer experience. Send feedback to sneha_jha@idgindia.com

We needed our system response times to complete within 10 milliseconds, 95-99 percent of the time—even while our platform processed up to 10,000 reads per second.