New purpose-built platforms are ready to leverage the Age of Instrumentation

Throughout history, humans have always created purpose-built tools that serve a specific need. Even in the Stone Age, humans shaped their stones to build a toolset to do different things. Sharp stones served as cutting tools, while larger hammerstones were used to strike or pound things.

Purpose-built tools are pervasive in every industry. We are most successful when we have the right tool for the job. Would you prefer that your surgeon use a Swiss Army knife to remove your inflamed appendix, or a precision-blade scalpel?

The information technology industry is rife with tools developed to serve a specific need. For example, customer relationship management (CRM) tools use a particular kind of database called a relational database to organize and correlate information about specific customers. Without the relational database, we’d still be tracking customer contacts in klunky spreadsheets… or Rolodexes! And just imagine trying to do anything in the Internet Age without a specialized tool called a web browser.

The same need for purpose-built tools exists in this new Age of Instrumentation. This wave sees organizations putting a vast series of sensors on things in order to take measurements or to detect the occurrence of recorded events. The “things” that are instrumented are physical, such as machinery on a factory floor, or virtual, such as software applications that run in containerized computing environments. Anything that can be instrumented for benefit will be instrumented, it’s just a matter of time.

Today sensors capture a stream of metrics such as temperature, pressure or even peak CPU utilization, or they can capture events such as accessing a file or the opening of a mechanical valve. This bit of data is time-stamped and collected into a database, where it joins millions of similar data points that together tell a story over time, or at a specific point in time. The ultimate goal is to interpret the meaning of those data points, make a relevant and real-time decision, and take some sort of immediate action to control what happens next.

For example, think about a collision avoidance system in a modern vehicle designed to prevent or reduce the severity of a collision. It uses correlated data coming from GPS, radar, LIDAR and camera systems – i.e., the sensors – to detect an imminent crash and warn the driver, or perhaps take autonomous action without driver input to apply the brakes or steer the vehicle in another direction. All of this happens in mere seconds. More importantly, the decisions the system makes matter in the moment, as this determines whether a crash can be avoided or not.

The purpose-built platform for the Age of Instrumentation

Like CRM, Internet browsing or even surgery, a system that captures and works with sensor-generated data requires a tool that is purpose-built for this activity. This type of platform has very explicit needs that can’t be properly addressed with general purpose tools. Here are the characteristics of a platform in the Age of Instrumentation:

• It must be able to handle incredible volumes of data—both in terms of ingesting the data into the system, and then storing it over time. Think about a self-driving vehicle. It’s covered in sensors that continuously measure all sorts of streaming metrics and observe countless events: speed, distance from the vehicle ahead, proximity to the painted lines on the road, status of traffic lights, and more. This huge volume of data being generated at very fast rates requires an infrastructure that can handle the immediate ingestion and storage of the data. Batch input won’t do. What’s more, the platform must be capable of compressing the data so that it doesn’t quickly consume all the storage capacity.
• The system must recognize time as a fundamental construct of the data. Every data point will have a time stamp so the system can understand precisely when something was measured or when an event happened. Applications that use the data need to support time-based functions, such as calculating the rolling average of the data, or comparing how a data point differs now compared to the same measurement taken in a different time period. Time is a fundamental constituent of any platform built for this purpose.
• The system must be able to down-sample the data; that is, get rid of some (but not all) of the data points after a period of time. It might be important to look at the freshest data at a very granular level now, but over time, the value of so much data diminishes. For instance, suppose you want to measure the consumption of network bandwidth. Today you want the data points at the millisecond level, but a month from now, it’s okay to drop many of the older data points so that you have only measurements that were taken once every second. The less acute data is fine for observing trends over time. What’s more, removing unnecessary data points saves on storage space.
• The platform needs to be able to deal with data in real-time. Consider that self-driving car. The system has to interpret and analyze the data in real-time in order to take action while it’s still meaningful. Waiting a few extra seconds to apply the brakes to avoid a collision is not viable; action has to be taken immediately, as soon as the data indicates a collision is imminent.
• And finally, the platform must be designed for control type functions. You want to use your critical time-stamped data in order to do things, such as apply the vehicle brakes to avoid a collision. Having visibility into a situation is useful only if you are able to control what happens next.

These fundamental technology requirements to leverage the Age of Instrumentation are a tall order. But new purpose-built platforms have been created to deal with these specific metric and event, or time-series, workloads and provide situational awareness to the business. These platforms support ingesting millions of data points per second, are able to scale both horizontally and vertically, are designed from the ground up to support real-time insights, and have strong machine learning and anomaly detection functions to aid in finding interesting business moments. In addition, they are resource-aware, applying compression and down-sampling functions to aid in optimal resource utilization, and are built to support faster time to market with minimal dependencies.

The Age of Instrumentation is here. The benefits to business are massive, but only if you use the right tools and right infrastructure to handle the new workloads. New workloads require purpose built infrastructure.