Making IoT Data Valuable

We're reminded almost daily about the great opportunities and possibilities the Internet of Things brings to the table. From railroad sensors that help prevent catastrophes to smartwatches that help us stay fit, more than 150 IoT devices get connected to the internet every second. But the data generated from IoT devices is only valuable if users can effectively analyze it, and performing that analysis presents its own set of challenges related to security, variety, and volume.

Data generated by IoT ecosystems will always be at risk – as the number of connected devices increases in organizations, it becomes more difficult for companies to keep security threats at bay.

Also, each connected device will create its own type of data. Getting value from it through traditional analytics encompasses a challenge on its own since most BI tools in the market have been designed to process structured data or to analyze data blanketed by proprietary semantic layers.

While implementing semantics on top of data is important, it has the potential to generate data chaos – each team within an organization will have different business interpretations of the same data, forcing analysts to spend more time reconciling results from different tools than gaining insights from their data.

Additionally, storing the massive amount of data generated by IoT devices calls for flexible and scalable cloud object stores. While object storage is a compelling solution due to its minimal cost, it's not something that BI tools can easily connect to. As a result, many companies aren't properly harnessing this data. Companies looking to unleash the value of IoT data need to take a holistic approach to act upon the opportunities that IoT brings across the entire business, as part of a transformation that relies fundamentally on analytics.

Regardless of the nature of the industry, to fully unleash the value of IoT data, companies need to develop operations around the following elements:

Harness and Capture the Right Data

Generating and capturing the right data for your Applications is the first step to creating a successful IoT ecosystem. Depending on the Applications, data needs to be managed and combined with other meaningful data such as customer, product or sales data, as well as environmental data. Processing these sources of data to gain valuable insights can only happen through a variety of analytical methods ranging from basic statistics all the way to sophisticated techniques such as machine learning.

Make Data Consumable

Temperature data, humidity data, heart rates, and logistics and tracking data, are just a few of the many types of IoT data that are often kept in cloud object stores such as Amazon S3. Buried in all that noise is the potential for amazing insights to drive quality, lower costs and improve operations outcomes.

Data-driven companies in every field are analyzing data looking for insights on how to improve their quality of products, increase service quality, reduce production downtime, increase sales and much more. But to accomplish this, the first step is to put useful data into the hands of analysts and BI consumers. And right now, IoT data, due to its volume, variety and the speed at which it's generated, isn't easily consumable. Whether it's a dataset for consumer tracking, noise levels or sales analysis, it takes weeks or months to just get data, before analysts can begin processing it.

Organizations using IoT data can accomplish an unlimited amount of innovative goals by getting their hands in the correct data at the perfect time. However, the process of building a proper data lake, ingesting data into it, curating it and securing it to make sure it lands in the right hands, is a complex and slow one. Many BI and data science tools currently in the market provide the means to easily analyze data, but to provide a real self-service experience, organizations have to make data easily accessible by curating it and properly cataloging it so consumers can find it.

Deliver Value by Embedding Into Existing Operations

While the quality of data and the insights obtained from it are highly valuable assets, one of the most important practices is to integrate IoT data into existing workflows and processes such as operations management systems for logistics and transport companies embedding insights into existing maintenance dispatch systems to prevent operations downtime, or hospitality and entertainment companies embedding customer behavior IoT data into sales and marketing operations systems to enhance consumer and visitor experience.

Organizations need to take a deep look into everything that's encompassed in their current architecture and define a way to integrate their existing technologies with their newly implemented IoT devices. The results from this kind of data blending can result in optimized lead generation processes, enhanced pricing and user experience and better machine efficiency and less downtime in production operations.

Data-as-a-Service

Data-as-a-Service puts together technologies needed to simplify the process of retrieving data from disparate sources, such as transactional databases, local files, and cloud stores. This vision not only helps to deliver valuable data to users in a timely and secure manner so they can start mining their data looking for clues to improve business decisions, but it also reduces the amount of grunt work that analysts need to perform before they can consume data.

Being able to make prompt business decisions on current data is key to succeed in competitive markets. Data-as-a-Service provide a self-service model that enables data consumers to explore, organize, describe and analyze data regardless of its location, size or structure, using their favorite tools such as Tableau, Python and R in a matter of minutes, while data engineers can dedicate their time to ensure integrity, security and governance without causing any delays in the data consumption process.

Embrace Open Source

In data analytics, the future is open source. Infrastructure based on open source delivers a number of benefits to enterprises, including faster development cycles (building on the work of the community of open source contributors), more secure and thoroughly reviewed code and no vendor lock-in.

For example, data infrastructure built on Apache Arrow allows enterprises to leverage the benefits of columnar data structures with in-memory computing providing dramatic advantages in terms of speed and efficiency.

Written by Tomer Shiran, co-founder and CEO of Dremio.