MQTT — Deep Dive

Message Queuing Telemetry Transport (MQTT) has been relevant for years. It’s enjoying even wider attention now thanks to the explosive growth of IoT, with both consumer and industrial economies deploying distributed networks, edge computing and data-emitting devices as part of everyday operations. This means lightweight, open-source and accessible protocols will only become more important with time. The article below provides a conceptual deep-dive into MQTT, how it works, and its uses now and in the future.

A Bit of Background on MQTT

MQTT is a publish-subscribe messaging protocol dating back to 1999 when IBM’s Andy Stanford-Clark and Cirrus Link’s Arlen Nipper published the first iteration. They envisioned MQTT as a way to maintain machine-to-machine communication on networks with limited bandwidth or unpredictable connectivity. One of its first Applications was to help keep sections of oil pipeline in contact with one another and with central sites using satellites.

As a result of the rugged conditions that inspired it, the protocol is lightweight and boasts a limited code footprint. This makes it ideal for low-power devices and those with limited battery lives, including now-ubiquitous smartphones and ever-growing numbers of sensors and connected devices.

As such, MQTT has become the go-to protocol for streaming data between devices with limited CPU power and/or battery life and for networks with expensive or low bandwidth, unpredictable stability or high latency. This is why MQTT is known as the ideal transport for IoT device data. MQTT is built on the TCP/IP protocol, but there's an offshoot, called MQTT-SN, which is designed for use on Bluetooth, UDP, ZigBee and other non-TCP/IP IoT networks.

MQTT isn’t the only publish-subscribe (Pub/Sub) realtime messaging protocol of its kind, but it has already achieved widespread adoption in a variety of surroundings that depend on machine-to-machine communication. Its contemporaries include Web Application Messaging Protocol, Streaming Text-Oriented Messaging Protocol, and Alternative Message Queuing Protocol.

For developers who want to build applications with robust functionality and wide compatibility with internet-connected devices and apps, including browsers, smartphones and IoT devices, MQTT is a logical choice.

How MQTT Works – the Basics

A communication system built on MQTT consists of the publishing server, a broker and one or more clients. The publisher doesn't require any configuration concerning the number or location of subscribers receiving messages. Likewise, subscribers don't need a publisher-specific setup. There may be more than one broker on the system distributing messages.

MQTT Basics

MQTT provides a way to create a communication channel hierarchy — sort of like a branch with leaves. Whenever a publisher has new data to distribute to clients, the message is accompanied by a delivery control note. Higher-level clients may receive every message, while lower-level clients may receive those relating to just one or two of the basic channel "leaves" at the bottom of the delivery hierarchy. These can facilitate the exchange of information as small as two bytes or as large as 256 megabytes.

An example of how you might configure an MQTT client to connect through an MQTT broker.

Here's the code (Gist link).

Any data published through or received by an MQTT broker will be binary encoded, as MQTT is a binary protocol. This means that you’ll need to interpret the message to get the original contents out. Here’s how it looks using Ably and JavaScript:

Here's the code (Gist link).

MQTT brokers may sometimes accumulate messages related to channels that have no current subscribers. If that’s the case, messages will either be discarded or retained, depending on instructions in the control message. This is useful at times when new subscribers may require the most recent recorded data point, instead of being forced to wait until the next dispatch.

Notably, MQTT uses plain text to transmit security credentials and doesn't otherwise provide support for authentication or security functionality. That’s where the SSL framework comes in, helping protect transmitted information from being intercepted or otherwise tampered with.

You can also use Ably token-based authentication with MQTT, if you want to avoid your API key being disclosed to the actual MQTT client at all (If you are using MQTT without SSL, then token authentication is mandatory, to prevent API keys being transmitted over cleartext.) An example of token authentication with MQTT:

Here's the code (Gist link).

MQTT Functionality – A Deeper Dive

According to IBM, MQTT is:

• Agnostic with respect to the content of the message.
• Ideal for one-to-many communication distribution and decoupled applications.
• Equipped with a “last will and testament” feature to notify parties of abnormal client disconnection
• Dependent on TCP/IP for basic connectivity purposes.
• Designed for at most once, at least once and exactly once message delivery.

To use MQTT, one can assume the role of publisher, consumer or both at once.

One of the distinguishing characteristics of MQTT is its leverage of channels. Other protocols utilize this in their own ways, but MQTT is unique in treating each one as a file path – like so:

Here's the code (Gist link).

Channels ensure that each listener receives the messages intended for them. MQTT accomplishes all kinds of useful communication functionality by treating channels as a file path, including allowing message filtration based on where — or at which level or branch — listeners subscribe to on the file path.

Understanding the MQTT Messaging Format

Here’s a look at the two components that comprise each message exchanged using the MQTT protocol:

Byte 1: Contains the message type (client request to connect, subscribe acknowledgment, ping request, etc.), a duplication flag, retain instructions for messages and information on the quality of service level (QoS).
Byte 2: Contains information on the remaining length of the message, including the payload and any data in the optional variable header.

The QoS flag in Byte 1 is worth special consideration since it lies at the heart of the variable functionality that MQTT supports. QoS flags contain the following values based on the intent and urgency of the message:

0 = at most once = server fires and forgets — messages may be lost or duplicated.
1 = at least once = recipient acknowledges delivery — messages may be duplicated, but delivery is assured.
2 = exactly once = server ensures delivery — messages arrived precisely once without loss or duplication.
Now, let’s look at the ways different QoS levels affect how MQTT can be used in IoT devices and other applications.

When Might You Use MQTT?

As IoT applications have found ways to disrupt industries on a huge scale, MQTT has come into focus as an open, straightforward and scalable way to roll out distributed computing and IoT functionality to a wider user base — in both the consumer and industrial markets.

As laid out above, MQTT is a lightweight messaging protocol built for questionable network quality and batteries with physical limitations, including power source and CPU. However, that doesn’t mean potentially lousy transmissions are its only application. From recurring data sampling to industrial machine control, MQTT delivers variable levels of service for a number of IoT infrastructure types:

Ambient sensor data: As mentioned, MQTT supports the “at most once” message delivery model. On networks with spotty coverage or high latency, this means information could be lost or duplicated. In fields where remote sensors record and transmit data at set intervals, this is a non-issue, since new readings are streaming in on a regular basis. Sensors in remote environments are usually low-power devices, making MQTT an ideal fit for IoT sensor buildouts with lower-priority data transmission needs.

Machine health data: In the interest of responding to emerging issues quickly and avoiding downtime, a wind turbine may require guaranteed delivery of machine health data to local teams even before that information hits a data center. For times like these, “at least once” message delivery ensures maintenance flags are seen by the right eyes at the right time to prevent problems, even if those flags may arrive as duplicates. This is for higher-priority machine-to-machine communication systems.

Billing systems: There are even higher-priority and higher-fidelity messages that require the right touch. In business situations where duplicated data entries are not okay, the exactly once QoS tag comes in handy. This eliminates duplicate or lost message packets across billing or invoicing systems and cuts down on anomalies and unnecessary strife with clients.

When Shouldn’t You Use MQTT?

Developers have a choice of protocols for designing and deploying bi-directional IoT functionality, including MQTT, HTTP, CoAP, WebSockets (where CPU/battery permits) and others. Whether MQTT is the best choice depends on what you want your application to do and the nature of your hardware.

Being designed for extremely low-bandwidth environments, MQTT can be rather inflexible in its quest to save every byte. For example, the spec defines five error responses with which a server can reject a connection (such as a bad username or password or an unacceptable protocol version). If the server wants to indicate some error that's not one of those five, it’s out of luck. Worse, if an error occurs once the connection is running, there's no mechanism for an error protocol message at all; the server can only shrug and abruptly close the TCP connection, leaving the client with no idea why it was dropped (and without any way to distinguish a deliberate disconnection from a transient network issue). For people used to more flexible and easier to debug (albeit less bandwidth-conserving) pub-sub protocols, such a miserly approach to bandwidth can be a little primitive.

MQTT is often referenced alongside HTTP, which is why Google mounted a study comparing the two for response time, data transmission size and other attributes of importance to developers. MQTT came out on top in Google’s tests but only when the connection can be reused to send multiple payloads.

HTTP and MQTT are both good choices for IoT applications because of their reasonably compact transmission sizes and their low strain on device battery life and memory.

CoAP is another protocol frequently compared to MQTT for IoT developments. The two are similar but come with notable differences. MQTT is a many-to-many protocol, whereas CoAP is mostly a one-to-one protocol for communications between a server and a client. CoAP also provides metadata, discovery and content negotiation features, which MQTT doesn't have.

In cases where clients need only to receive data, Server-Sent Events is also a valid choice.

How to Quickly Get Set up With MQTT

The MQTT GitHub repo has an extensive list of open source MQTT libraries across various languages. Below are two examples of getting set up with an open-source MQTT broker and JavaScript and .NET library.

Eclipse Mosquitto – An Open-Source MQTT Broker

Eclipse Mosquitto is an open-source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers.

MQTT.js

MQTT.js is a client library for the MQTT protocol, written in JavaScript for node.js and browsers. Here’s an example of sending a message using MQTT.js:

Here's the code (Gist link).

MQTTnet

MQTTnet is a high-performance .NET library for MQTT based communication. It provides a MQTT client and a MQTT server (broker).

Set up an MQTT client:

Here's the code (Gist link).

After setting up the MQTT client options a connection can be established. The following code shows how to connect with a server:

Here's the code (Gist link).

Consume incoming messages:

Here's the code (Gist link).

Publish a message:

Here's the code (Gist link).

Service providers at the enterprise level have MQTT-ready servers available that support scalable messaging between mobile applications, industrial machines and a wide variety of other IoT Applications.

What About Deploying MQTT at Scale?

There are two considerations when it comes to MQTT and scale: is it the right protocol and are the infrastructure and network capabilities needed to handle increased device-to-device streaming over MQTT.

Lightweight Machine-to-Machine (LWM2M) is another protocol to consider alongside MQTT at the enterprise level. Compared with MQTT, LWM2M is sometimes better suited for longer-term IoT deployments. MQTT is ideal when companies want to engage in a trial run of IoT functionality without a lot of buy-in, whereas LWM2M provides features for companies building longer-term infrastructure that’s versatile. LWM2M also provides superior device management tools, such as connectivity monitoring, firmware updates and remote device actions. For enterprises with lots of unmanaged devices sending high volumes of data to a central IoT platform, LWM2M is the better choice. However, we’re talking huge-scale IoT deployments, so, most of the time, MQTT is a more than adequate choice. Plus, of the two, MQTT remains more common with wider support.

Then there are infrastructure capabilities. The number of concurrent connections a server can handle is rarely the bottleneck when it comes to server load. Most decent MQTT servers/brokers can support thousands of concurrent connections, but what’s the workload required to process and respond to messages once the MQTT server process has handled receipt of the actual data? Typically there will be all kinds of potential concerns, such as reading and writing to and from a database, integration with a server, allocation, management of resources for each client and so on. As soon as one machine is unable to cope with the workload, you’ll need to start adding additional servers, which means now you’ll need to start thinking about load-balancing, synchronization of messages among clients connected to different servers, generalized access to client state irrespective of connection lifespan or the specific server that the client is connected to – the list goes on and on.

Such concerns are deserving of an article all of their own, and you’ll find plenty in the Engineering section of the Ably blog – see in particular a write-up on some of the complexities of operating real-time messaging infrastructure at scale.

Where Does MQTT Stand today?

In April 2019, OASIS officially released v5.0 of MQTT as an official open-source standard. OASIS is a nonprofit consortium consisting of 600 member organizations and 5,000 individual participants.

Version 5.0 introduces several new features that should be of interest to realtime developers. These new features are backward-compatible with current versions of MQTT already in deployment.

Those new features include:

• Improved reporting for errors: Return codes can now inform users when data isn’t transmitting successfully for whatever reason. Reason strings are supported, but optional, and help improve diagnostics and troubleshooting efforts.
• Subscription sharing: To help balance the load, subscriptions can be shared across multiple clients on the receiving end.
• Message properties: Version 5.0 introduces metadata as a part of the message header. It can convey additional information to the end-user or it may facilitate some of the other features included below.
• Channel alias: Publishers can replace channels with numerical identifiers to cut down on how many bytes require transmission.
• Message expiration: Messages can be tagged for automatic deletion if the system cannot deliver them within a preset amount of time.

The full list of new functionality MQTT 5.0 can be found in Appendix C of the official standard.

In addition to the panoply of consumer-level devices and services on the market today, MQTT has found a home in enterprise infrastructure of all shapes and sizes. This includes smartphones and tablets, energy-monitoring systems, medical devices, oil rigs and drilling sites, the automotive and aerospace industries and sensors and machine vision systems used in material handling, construction, supply chain, retail and more. The future looks bright for IoT solutions that leverage MQTT.

Written by Eve Harris at Ably Realtime.
This post was originally published at ably.io.