The gathered crowd looked on with amazement and let out an audible gasp. The volunteer thought he had stumped the magician with his chosen number, 83. The magician had written 16 numbers in a four by four matrix \u2014 and 83 was nowhere in the mix.\nBut then the magician broke into a wide grin.\n\u201cCheck this out. If you add up all the numbers across each row, they add up to 83. The numbers in each column? That\u2019s right, 83. In fact, every combination adds up to your number. Amazing, right? It\u2019s just like what happens when you get data right \u2014 it\u2019s like magic!\u201d\nThe magician was pitching for Diwo, a new cognitive decision-making platform, at their booth at the Strata Data Conference held last week in New York. It was a fun way to introduce their new solution \u2014 and, perhaps unwittingly, it was also allegorical to the evolution of the big data market.\nWhile we have been talking about big data, data science and analytics for quite some time, there is an evolution of the market that was on full display at this year\u2019s event. It could be seen in the several interwoven themes that permeated both the keynote stage and the exhibit floor. The overarching message: It\u2019s time to get big data right.\nThese themes all touched on the same broad idea that it is time to move beyond the exploration stage and apply big data, data science and analytics in real life and at scale so that the power of data can transform business models and the customer experience \u2014 and perhaps make it all feel a bit like magic.\nMaking big data real\nFor most of its existence, big data has been a technically focused domain. While the business implications have (almost) always been clear, the focus of the market has predominately been on experimentation and figuring out how to solve the big, hairy technical problems incumbent with massive datasets.\nThere were, of course, successful applications of big data that produced significant business results, but the primary driver of the market was technical development \u2014 not business application. This year, however, there was a visible change.\nFirst, there was significant airtime given to the societal impact of data and the important role that data scientists and practitioners must play as the industry continues to evolve.\n\u201cWe haven\u2019t established standards for what is good enough in data science,\u201d cautioned Cathy O\u2019Neil, mathematician and author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. \u201cBad algorithms can destroy people\u2019s lives\u2026and AI isn\u2019t fair. It automates the status quo and all of its implicit biases. That needs to be acknowledged, particularly when we\u2019re focused on those things that impact people\u2019s lives.\u201d\nManuel Garc\u00eda-Herranz, chief scientist with UNICEF\u2019s Office of Innovation, and Jer Thorp, innovator in residence at the Library of Congress, echoed similar sentiments about both the positive \u2014 and potentially negative \u2014 impact that data can have on the world.\nGarcia-Herranz shared how UNICEF is integrating data science and real-time systems to discover insights while it can still use them to take meaningful action and \u201capply data for humanity.\u201d\nThorp, on the other hand, cautioned attendees that their desired objectivity could keep them from understanding the real power and impact of data. His advice was as simple as it was profound: \u201cGet out of your chair.\u201d\nAt the same time, the intersection of big data and artificial intelligence (AI) is increasing the urgency with which enterprises are addressing their big data initiatives. While big data's nature as the fuel that powers AI has made the two symbiotic, the rapid ascent of AI as a top-of-mind issue with business executives is making the need to deploy big data on an enterprise scale a strategic imperative.\nMaking big data real-time\nShifting the focus to how organizations apply big data in the real world has also led to a second significant trend: the shift to apply data and analytics in real time.\nThere is increasing recognition that post-transaction analytics is not the only big data use case \u2014 and in fact, may not even the best one. Many organizations now realize that they can harness the greatest value from their big data initiatives by applying the resulting analytics and insights at the point of the transaction.\nWith this use case, organizations can go beyond merely using data retrospectively for analysis and planning and can instead use it to shape the customer experience, enable better decision-making and reduce risk in real time before negative outcomes occur. This use of data at the point of transaction can take many forms and should be an essential element of any modern big data strategy.\nSeveral technology companies have introduced tools and strategies to help enterprise organizations integrate the insights and analytics they derive from their big data initiatives in real time. These include (in alphabetical order):\n\nCambridge Semantics: An end-to-end, exploratory analytics solution built on a semantic relational data model that enables real-time analytics and reduces time to market by structuring data according to business context.\nMapR: A converged data platform offering what it calls a "data fabric" that integrates traditional data lakes with streaming data in a single, location-independent, and context-aware platform.\nSplice Machine: An application development platform that creates a new breed of "predictive applications" that merge transactional and analytical processing and injects analytics-derived insights into the application workflow.\nStriim: A real-time data integration and streaming analytics platform that analyzes data at the time of ingestion to support decision-making with real-time insights.\nVoltDB: An operational data platform that offers real-time event processing and analytics with millisecond response times.\n\nWhile these technology providers are taking very different approaches to apply big data in real time, each of them is delivering the same broad message to enterprise organizations: The best time to use big data is right now.\nMaking big data work\nUnderstanding the real-world implications of big data and moving its application to the point of a transaction won\u2019t do any good, however, if an organization is unable to make big data work at enterprise-scale and within the enterprise operating model. As organizations have attempted to move big data beyond the realm of experimentation and head toward full-scale, enterprise-wide application, they have run into significant governance, management, and scale issues.\nIt should be no surprise, therefore, that the third major theme on display at Strata was the need for organizations to make big data work at enterprise-scale. Unquestionably, a significant part of that process requires cultural and organizational transformation. But it\u2019s also clear that organizations must also transform how they apply the technology itself to make it work at scale.\nSeveral technology companies showed new technologies and technology-driven approaches that they believe will help enterprise organizations grapple with the various aspects of making big data enterprise-ready, including (in alphabetical order):\n\nDataguise: A data governance platform that discovers, detects, protects and monitors sensitive information, such as PII, PCI, and HIPPA, wherever it is within the organization\u2019s data landscape.\nDataiku: A big data platform that creates a single data pipeline that breaks down silos and enables data scientists and data analysts to easily work together and speed the deployment of predictive solutions.\nDriveScale: A software company changing the way organizations deploy and consume storage with what it calls "software composable infrastructure," which connects disaggregated components (compute and storage) in an intelligent and highly dynamic manner.\nPure Storage: A highly efficient storage platform purpose-built for scaled-out big data deployments, modern analytics needs, and AI.\nZaloni: A data lake management platform focused on helping enterprises operationalize their data lake and rapidly get to business value at scale.\n\nGetting big data right\nThe technical challenges that have been at the heart of the big data industry persist \u2014 and will endure for the foreseeable future. As data growth continues exponentially, enterprise organizations and the technology companies that serve them will remain locked in a continuous battle to tame it and make it manageable.\nNevertheless, it is clear that organizations are now beginning to earnestly address the challenges that come with applying big data across the enterprise so that they can accelerate digital transformation and feed their growing AI initiatives.\nAnd, it seems clear, both the industry and enterprise organizations are recognizing that for big data to fulfill its promise, they need to get big data right.\nDisclosure: The Strata Data Conference provided me a free pass for this event, a standard industry practice.