Machine Learning and Azure Sphere

The Internet of Things (IoT) is arriving at pace, enabling new applications and business models across many consumer, enterprise, and industrial sectors. When you attach a “thing” to the Internet, you connect new data sources, which can be costly and resource-intensive to store or upload. As IoT continues to grow, organizations and users alike are finding immense benefit in applying IoT paired with Machine Learning and Azure Sphere.

ML is excellent at processing noisy and complex sensor data to determine higher-level insights like “operating normally” versus “cooling unit failed” in a vibration sensor data stream, or “wake word detected” in an audio stream, or “items placed correctly” versus “item out of place” in a video stream.  

ML can be used on the IoT device itself, in the cloud, or a combination of both.  ML on IoT devices brings the processing closer to the data generation. This has lots of benefits, including (a) internet connectivity is not relied upon for the higher-level states to be determined and actions to be taken on that basis, (b) fewer resources are consumed for transmission of data to the cloud, and data that is not needed for the long-term can be immediately used and then deleted rather than stored, (c) it can enhance privacy because the raw data might include more personal information (such as voices accidentally captured by an audio sensor designed to listen to machine behavior), so if that data is processed and discarded locally, then there are fewer privacy risks.

Of course, there are also benefits to operating ML in the cloud. The availability of plentiful server resources means that cloud-based ML can be faster and use more complex models to achieve higher accuracy. Retraining of models, which is resource-intensive, can be quicker, and redeployment of models can be immediate.

Hybrid designs are also possible, which involve ML on IoT devices and in the cloud. One example most of us will recognize is voice assistants—the wake word is recognized locally on the device. Still, then the voice data is streamed to a cloud service where language understanding and response generation is done.

Azure Sphere and Machine Learning

You can use Azure Sphere to build IoT devices while relying on Microsoft to stay on top of OS security threats, which means that you can focus on the actual application logic that defines your device, including ML where appropriate.

There are lots of advantages to using Azure Sphere for ML applications.

• First, Microsoft secures the device over its full lifetime. Since many devices requiring ML include sensors such as cameras or microphones, it is important to invest in ongoing security. This is a prerequisite for privacy—an insecure device cannot protect private data.
• Second, Azure Sphere provides a unique identity for the device and a certificate that encodes that identity for connection to cloud services. The result is that no other device can pretend to be this device. This is particularly important for ML applications because data that is falsely regarded as coming from a device can pollute training datasets, degrading ML performance.
• Third, Azure Sphere’s application update service provides an easy way to update the ML model and ML runtime.
• Finally, ML models are valuable intellectual property, and the security of the Azure Sphere platform protects those models.

Azure Sphere supports machine learning through “Tiny ML” and detailed four different Tiny ML solutions that can run on the MT3620. This post will describe a demo that uses one of these ML solutions in combination with cloud ML service, showing how easy it is to build such hybrid ML solutions with Azure Sphere.

Recognizing Faces: Machine Learning and Azure Sphere to Enable Cognitive Services

To illustrate how Azure Sphere-based ML and cloud ML can work together, we’re going to use an example application that recognizes individuals. This could be used for scenarios like appliances that personalize their experience based on the user or safety applications like only allowing authorized and trained individuals into potentially dangerous locations in an industrial building.

Face recognition is supported by an Azure Cognitive Service. It is quick and easy to train this cognitive service to recognize individuals. However, if this were to be used directly by an IoT device watching for a user, then that IoT device would have to upload pictures constantly, which is a poor design for many reasons including the bandwidth and cloud services costs, privacy implications, and energy consumption.

On the other hand, deploying this machine learning model to MT3620 presents difficulties. An appropriate ML model would need to be retrained for each device's individual user, meaning that every IoT device would need a different model. Some ML models are large, and while various techniques can be used to compress them, ultimately, this may not be achievable with high performance for some tasks.

The solution is to use the hybrid approach. In our face recognition example, we perform person detection on MT3620 using a pre-existing model provided by Mediatek as part of their Neuropilot-Microframework. Only when a person is sensed, do we need to use the Cognitive Service for face verification to determine identity.

Demo 

The following video shows the demo in action:

We built out this demo using the same person detection hardware and NeuroPilot Micro ML model documented by Mediatek. To use the Cognitive Services Face API, this webpage describes how to get started, and includes links to samples for that API.

Responsible AI

Finally, we should note that machine learning is a technology that is particularly important to apply responsibly. ML-based products should be designed to protect principles such as being inclusive, human-centric, behaving understandably, and promoting fairness.