Built with BigQuery: How BigQuery Helps Leverege Deliver Business-critical Enterprise IoT Solutions at Scale

Introduction

Leverege is a software company that enables market leaders around the globe to quickly and cost effectively build enterprise IoT applications to provide data-centric decision capability, optimize operations, improve customer experience, deliver customer value, and increase revenue. Leverege’s premier SaaS product, the Leverege IoT Stack, runs natively on Google Cloud and seamlessly integrates with Google’s vast array of AI/ML products.

Leverege uses BigQuery as a key component of its data and analytics pipeline to deliver innovative IoT solutions at scale. BigQuery provides an ideal foundation for IoT systems with its data warehousing capabilities, out-of-the-box data management features, real-time analytics, cross cloud data integration, and security and compliance standards. These features enable customers to easily integrate data processes and use the resulting datasets to identify trends and apply insights into operations. 

Context and IoT Industry Background

The Internet of Things (IoT) connects sensors, machines, and devices to the internet, allowing businesses in every industry to move data from the physical world to the digital world, on the edge, and in the cloud. The adoption of large-scale IoT solutions gives businesses the data they need to improve efficiency, reduce costs, increase revenue, and drive innovation. 

The power of IoT solutions, and their impact on the global economy, are driving demand for robust and secure enterprise data warehouse capabilities. IoT presents a particular challenge on the infrastructure level because many technical requirements at scale cannot be predicted in advance. Some customers need to manage massive IoT datasets while others require real-time data streaming or fine-grained access controls. 

The breadth of infrastructure requirements in the IoT space means Leverege depends on partnering with a best-in-class cloud computing provider. On the technical side, a full-featured data warehouse is required to meet customer needs and bring them to scale. On the financial side, the end-to-end solution must be designed to manage and reduce overall costs, accounting for each of the solution’s components (hardware, connectivity, infrastructure, and software).

By leveraging the scalability and flexibility of Google Cloud Platform and BigQuery, Leverege’s customers can affordably store, process, and analyze data from millions of connected devices and extract the value they need from sensor data.

Introduction to Leverege Using Google Cloud

Leverege offers a customizable multi-layer IoT stack to help organizations quickly and easily build and deploy IoT solutions that provide measurable business value. The Leverege IoT Stack consists of three components:

• Leverege Connect is focused on device management, enabling the secure provisioning, connection, and management of distributed IoT devices. Leverege Connect serves as a replacement for Google IoT Core which will be retired in August 2023 and supports protocols such as MQTT, HTTP, UDP, and CoAP.
  
  
• Leverege Architect is focused on data management, enabling the ingestion, organization, and contextualization of device and business data with the ability to apply AI/ML for powerful insights and/or expose via APIs to external services.
  
  
• Leverege Build optimizes application development, enabling the generation, configuration, and branding of end-user applications with tailored experiences on a per-role basis; all with no-code tooling.

Leverege IoT Stack is deployed with Google Kubernetes Engine (GKE), a fully managed Kubernetes service for managing collections of microservices. Leverege uses Google Cloud Pub/Sub, a fully managed service, as the primary means of message routing for data ingestion, and Google Firebase for real-time data and user interface hosting. For long-term data storage, historical querying and analysis, and real-time insights, Leverege relies on BigQuery.

Leveraging BigQuery to Deliver and Manage IoT Solutions at Scale

Use Case #1: Automating Vehicle Auctions for the World’s Largest Automobile Wholesaler

The world’s leading used vehicle marketplace faced the costly challenge of efficiently orchestrating and executing simultaneous in-person and online car auctions on parking lots up to 600 acres in size. Before the IoT solution was deployed, manually staging thousands of vehicles each day involved hundreds of people finding specific vehicles based on hard-to-discover information and attempting to arrange them in precise order. This manual process was highly inefficient, unreliable, and negatively impacted the customer experience since vehicles routinely missed the auction or were out of sequence. 

To solve the problem, the customer built low-cost, long battery life GPS trackers and placed them inside all of the vehicles on the lot. Leverege integrated the devices into a holistic end-to-end solution, providing full awareness and visibility into precise car location, diagnostics, automated querying, analysis reports, and movement with walking directions to vehicles of interest. This digital transformation saved the customer millions of dollars a year while simultaneously increasing customer satisfaction by a significant amount.

After the solution scaled nationwide, monitoring the health of the devices and system was paramount for operational success. BigQuery data partitioning and autonomous analysis jobs allowed for a cost effective way to manage and segment system alerts and reports of overall system health using very large datasets.

Use Case #2: Analyzing the State and Readiness of Boats Anywhere in the World in Real-time 

Working with the largest boat engine manufacturer in the world, Leverege delivered an IoT solution providing boat owners and fleet managers with real-time, 24/7 access to the state, readiness, and location of their boats around the globe.

Seamlessly and reliably providing real-time marine data to boat owners requires technical integration across hardware, software, and connectivity, a problem uniquely suited for an IoT solution. The customer’s “Connected Boat” product reports a high volume of disparate data including the status of every electrical, mechanical, and engine subsystem. Some of this data is only important historically when incidents and issues arise and boat owners need to investigate. 

BigQuery allows Leverege to record the full volume of historical data at a low storage cost, while only paying to access small segments of data on-demand using table partitioning. 

For each of these examples, historical analysis using BigQuery can help identify pain points and improve operational efficiencies. They can also do so with both public datasets and private datasets. This means an auto wholesaler can expose data for specific vehicles, but not the entire dataset (i.e., no API queries). Likewise, a boat engine manufacturer can make subsets of data available to different end users.

Leverege IoT Stack Reference Architecture: Integrating Components to Deliver Robust, Scalable, and Secure Solutions 

The Leverege IoT Stack is built on top of Google Cloud's infrastructure, making use of several core components that work together to deliver a robust, scalable, and secure solution. These components include:

• GKE: Leverege uses GKE to deploy a collection of microservices and easily scale end-to-end IoT solutions. These microservices handle tasks such as device management, data ingestion, and real-time data processing. In addition, GKE secures a high degree of business continuity and enables self-healing and fault tolerance, which allow Leverege to provide enterprise-grade availability and uptime. These capabilities are crucial for Leverege to meet requirements specified by Service-Level Agreements.
  
  
• Pub/Sub: Leverege uses Pub/Sub to orchestrate the routing of messages for data ingestion, allowing customers to process data in near real-time. This provides a highly auto scalable, fault-tolerant message queuing system.
  
  
• Firebase: Leverege uses Firebase for real-time data and UI hosting, providing customers with a responsive and interactive user experience. With Firebase, customers can easily access and visualize IoT data, as well as build and scale applications with minimal effort.

BigQuery: BigQuery is a fundamental part of the Leverege solution. It enables customers to run long-term data storage and complex, historical SQL-like queries. These queries can be run on large amounts of data in real-time, providing customers with actionable insights that can help improve operational efficiencies.

Solution: Leveraging Core BigQuery Features for IoT Use Cases 

Many technology companies make extensive use of specific BigQuery features to deliver business-critical outcomes. Some use cases demand sub-second latency; others require adaptable ML models. By contrast, enterprise IoT use cases typically include a broad set of requirements necessitating the use of the full breadth of BigQuery’s core features. For example, Leverege uses an array of BigQuery features, including: 

• Data Storage: BigQuery serves as a limitless storage platform allowing Leverege customers to store and manage large-scale IoT data with high availability, including real-time and historical data. Some of Leverege’s integrated devices can report thousands of times a day. At a scale of millions of devices, Leverege’s customers need a scalable data warehouse.
  
  
• Real-Time Streaming: BigQuery also provides a powerful streaming capability, which allows the Leverege IoT Stack to ingest and process large amounts of data in near real-time. This is crucial to components of Leverege Build, which offers out-of-the-box charts and graphs using historical data. These tools are more valuable with the integration and use of real-time data. Streaming capabilities ensure customers easily access full-scope data without searching Google Firebase.
  
  
• Data Partitioning: BigQuery enables cost-effective, fast queries by providing customizable data partitioning. The Leverege IoT stack partitions nearly all historical tables by ingestion time. Because most internal history queries are time-based, this results in significant cost savings.
  
  
• Data Encryption: BigQuery provides built-in encryption at rest by default, allowing customers to securely store sensitive data and protect it against unauthorized access.
  
  
• Access Control: BigQuery provides numerous secure data sharing capabilities. Leverege uses linked data sets and authorized views with row level policies to enforce strict access control. These policies are critical because many IoT projects allow for multi-tenancy and data siloing.
  
  
• Data Governance: BigQuery provides a robust set of data governance and security features, including fine-grained access controls, which Leverege uses to enforce intricate access control policies down to the row level.

In addition to BigQuery’s core features, Leverege uses BigQuery Analytics Hub private data exchanges and Authorized Views on Datasets provides distinct advantages over old methods (e.g. CSV exports and FTP drops). Authorized Views on Leverege's BigQuery datasets allow for intricate access policies to be enforced, while also providing Leverege’s customers the ability to analyze data using tools like Looker. Using these BigQuery features, Leverege can give customers controlled and metered access to source data without providing direct access. This feature is fundamental to meeting governance requirements across the enterprise.

BigQuery's built-in machine learning capabilities also allow for advanced analysis and prediction of trends and patterns within the data, providing valuable insights for our customers without moving the data to external systems. Furthermore, the ability to set up automatic data refresh and materialized views in BigQuery ensures that our customers are always working with the most up-to-date and accurate data by getting better performance and reducing unnecessary costs.

Benefits and Outcomes

Google Cloud infrastructure and BigQuery features enable Leverege to provide a highly scalable IoT stack. In IoT, the central challenge isn’t deploying small-scale solutions; it’s deploying and managing large-scale, performative solutions and applications by scaling in a short span of time without rearchitecting. 

BigQuery table partitioning splits data into mini tables divided by an arbitrary time range. For many Leverege customers, data is divided by day and enforced when querying data through the Leverege IoT Stack. Partitioning data tables by time range guarantees queries are restricted to a small subset of data falling within the targeted time range. By using partitioning, Leverege can deliver a performant solution at minimal cost. 

BigQuery clustering further enhances performance by splitting data into designated fields. To make queries more efficient, Leverege uses clustering to query data that meet pre-designated filter criteria. In a large-scale solution with 100,000 devices, Leverege can cluster data tables and query the history of single devices, greatly accelerating searches and making the system much more performant. In addition, the reclustering happens seamlessly in the background without any extra costs.

The integration of the Leverege IoT Stack and Google Cloud, including BigQuery, today power business-critical enterprise IoT solutions at scale. The continued rapid pace of development on the infrastructure and application levels will be essential in delivering the next generation of IoT solutions.

Learn more about Leverege’s capabilities or to request a demo.

The Built with BigQuery Advantage for ISVs

Google is helping tech companies like Leverege build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs through the Built with BigQuery initiative, launched in April as part of the Google Data Cloud Summit. Participating companies can: 

• Get started fast with a Google-funded, pre-configured sandbox. 
• Accelerate product design and architecture through access to designated experts from the ISV Center of Excellence who can provide insight into key use cases, architectural patterns, and best practices. 
• Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies with the portability and extensibility they need to avoid data lock-in. 

Click to learn more about Built with BigQuery.

Written by Dr. Ali Arsanjani, Director, Cloud Partner Engineering, Google and Justin Mikolay, Senior Vice President for Business Development, Leverege