Innovation can pay off in a big way, but can you afford it?\nTo decide if you should try an innovative approach, you naturally weigh the large, potential benefits against the known risks -- the disruption of existing processes and the added cost of resources needed to carry out the new project. A common response to these risks is to avoid uncertainty as much as possible and only try \u201csafe\u201d projects using familiar approaches. But innovative projects are speculative \u2013 they may fail. And in business, a \u201cno failure\u201d policy is also a \u201cno innovation\u201d policy. That means you could miss out on the big wins that innovation can deliver.\nYou can improve your tolerance for a project that may fail if you find ways to lower entry costs while protecting existing SLAs. But you may unintentionally have baked in higher costs and risks than necessary in the way that your data infrastructure and processes work. Let me tell you a story that shows how getting these right can allow you to take on speculative projects that may ultimately lead to high rewards.\nReal world example: taking a risk can be worth it\nAbout eight years ago, a large, well established financial company stepped outside of their comfort zone and tried an innovative project built on an AI-based system. After both the merchant and the customer opted in, the new application would provide targeted discounts and other types of upselling offers.\nFrom the very beginning, there was high uncertainty. How long would the application take to develop? And after all the development work, would the application actually drive net new revenues? The financial company took the risk, and the entire system was developed, tested, and put into production in a matter of months.\u00a0\nThe new application quickly paid off with significant new revenue streams. This innovation was a big win--yet, with all the uncertainties, how did this financial company possibly afford to do the experiment?\nThe answer lies in the way they could minimize costs and risk, thanks to the flexibility and efficiency of their data infrastructure. They kept costs down in part because their data infrastructure did not require them to build a new cluster; it could easily support the additional project. In addition, they used customer transaction data that was already being collected for their mainstream business processes, so it wouldn\u2019t take more effort or resources. And lastly, they did all this in a way that didn't interfere with the primary purpose of the cluster or encroach on critical business goals. In short, they were able to afford to experiment because their approach bounded the risk and cost of experimentation.\u00a0\nLet\u2019s look more concretely at strategies for system design and data infrastructure that make it easier for you to lower entry costs and bound risk when you try new approaches.\nStrategies to make innovation affordable\nTo afford innovation, you must also be able to afford to fail. Consider these four strategies to make it reasonable to try innovative approaches:\n\nTake advantage of sunk costs\u00a0\n\nMany of these experimental projects require very large datasets. If your data infrastructure and system design require you to collect or copy new data or build a new cluster from scratch for new projects in order to protect existing SLAs, the entry costs will be too high. You would have to guarantee a successful outcome and won't be able to tolerate failure. That, in turn, means you can\u2019t afford to try something that is actually innovative.\u00a0\nAnother way to take advantage of sunk costs is to use a secondary cluster normally intended for\u00a0disaster recovery as a sandbox for experimental projects. You have a comprehensive copy of data standing ready in case of catastrophic loss at the primary data center. Why not put it to work in the meantime as a resource for new project development?\u00a0\n\nMake use of robust multi-tenancy\n\nTo take advantage of sunk costs and reduce risk of disruptions by new projects, your data platform needs to support real multi-tenancy, making it easy and safe for multiple applications and users to share the same data. For this to work, you need easily managed and consistent fine-grained access control. You also need to provide efficient automated resource allocation and be able to use containerized applications. If you can do this,\u00a0you can attempt a new project with an uncertain outcome but with a potentially high return -- without excessive entry costs or the danger of interfering with critical SLAs.\n\nHandle data logistics at the platform level, not the application level\n\nYou can also reduce entry costs for new projects if you handle data logistics at the platform level, not the application level. Not only is this approach generally more efficient and less error prone, it also means your developers don\u2019t have to re-implement everything as they build a new project. In other words, doing logistics at the platform level is another way you truly take advantage of sunk costs.\u00a0\nAnother real world example illustrates this strategy. A retail company with highly distributed online services needed to move telemetrics from many edge data sources to their core data center to analyze service quality and for billing. This data movement had previously been done at the application level, but that was cumbersome and imposed a heavy burden on developers.\nBy switching to use the event message stream capabilities built into the platform level, data acquisition and transfer to core was implemented easily and reliably. This change to platform-based logistics freed up developer time for other projects to the point that the customer claimed a net result of \u201ca year of developer time in the bank\u201d.\u00a0Details of this interesting use case are found in the article \u201cUsing Data Fabric and Kubernetes in Edge Computing\u201d.\u00a0\n\nExpand your idea of what constitutes a deliverable\n\nA final strategy is to re-think what constitutes a \u201cdeliverable\u201d. If a project must deliver a successful production system, the risk of failure may preclude trying anything new. What if you also count experience as a deliverable? Of course, you still must impose specific limitations on resources and time allotted to experimentation. Additionally, you will need a well-defined target that aligns with business goals. But the value of experience\u2014if acknowledged and communicated across your organization\u2014can improve the outcomes for other new projects.\nThis last idea underlines the importance of making learning a key part of your project. Innovation relies on an organization-wide build-up of tribal knowledge and experience that eventually reaches critical mass to create a solid competitive edge versus your competitor.\nKey Enablers\nSeveral key enablers will help you put these strategies into play.\u00a0Both of the real world examples described here were MapR customers (MapR was acquired by HPE in 2019) and used HPE Ezmeral Data Fabric (formerly known as the MapR Data Platform). The HPE Ezmeral Data Fabric lets you handle data logistics efficiently at the platform level with conventional access to a highly scalable file system. It also supports bi-directional replication of built-in tables and event streams, incremental mirroring across data centers from edge to core to cloud, and true multi-tenancy.\nAnother enabler to support multi-tenancy and make innovation affordable is to leverage the convenience, resource utilization optimization, and improved performance of a container platform. HPE recently announced general availability of the HPE Ezmeral Container Platform that used the HPE Ezmeral Data Fabric as its data layer. In a March 2020 CIO.com article, Robert Christiansen talks about the benefits IT teams enjoy by \u201ccollapsing the stack\u201d and improving the containerization experience through use of the HPE Ezmeral Container Platform.\nNext Steps\nA good starting point is to look at your system design: consider structural changes to methods and data infrastructure that could make your system better able to support multi-tenancy and avoid unnecessary costs and risks for trying new projects. Then explore your data and your team\u2019s ideas to identify new projects\u2014even speculative ones\u2014with high potential rewards in line with your business goals.\n____________________________________\nAbout Ellen Friedman\n\nEllen Friedman is a principal technologist at HPE focused on large-scale data analytics and machine learning. Ellen worked at MapR Technologies for seven years prior to her current role at HPE, where she was a committer for the Apache Drill and Apache Mahout open source projects. She is a co-author of multiple books published by O\u2019Reilly Media, including AI & Analytics in Production, Machine Learning Logistics, and the Practical Machine Learning series.