Reduce Message Payload in an IoT Setup

Using Protobuf with Angular2 and MQTT

In this article I’m writing about the usage of protobuf with typescript in Angular 2 via MQTT. The repository for this example is here

Protobuf

Protobuf a short for Protocol Buffers are

[…] Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Regarding a usecase, think of sending data from a small device, e.g. an arduino from somewhere remote via mobile network. In this case every byte counts.

MQTT

MQTT is

A lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks

Angular2

To keep things simple, I’ll use Angular2, one of many really nice frontend frameworks to speed up developing a small example for this article.

Using Protobuf

Google did a great job and installing/using protobuf is dead simple. Either you clone the repository and build it fro yourself, or download the suiting prebuilt library for your system and your desired target language. Googles protobuf support as target languages C++, C#, Java, JavaScript, Objective-C, PHP, Python and Ruby. There are also several other community implementations for other languages, like Haskell, C or a very nice all-you-need-project protobuf.js.

For every supported language, there is a folder in the main project, containing a readme file how to use the compiler to compile a *.proto. Proto files are the definitions of your messages you want to serialize/deserialize. To read more about the format, I can recommend the Language Guide (proto3).

To keep this example simple, we just want to send an uint64 via mqtt. Here is the protobuf file value.proto

syntax = "proto3";

package example;

message Value {
  uint64 value = 1;
}

To actually use it, we have to compile it to a javascript file. As I mentioned before, the official protobuf compiler has a js target with different options. Unfortunately, it’s difficult to use with typescript, which I want to use.

protobuf.js for the rescue!

Along the library files you have to include in your project to encode or decode protobuf messages in general, protobuf.js has a command line interface to directly create the needed javascript files for our value.proto file and the typescript definitions without the need of protoc from google/protobuf.

To compile the value.proto file and directly place it where we want to use it in our example, execute the following script:

./node_modules/protobufjs/bin/pbjs -t static-module -w commonjs -o ./src/protobuf/value.protobuf.js ./value.proto

protobuf.js additionally has a tool to generate the typescript definitions for us, so using it in Angular2 is very convenient.

./node_modules/protobufjs/bin/pbts -o ./src/protobuf/value.protobuf.d.ts ./src/protobuf/value.protobuf.js

To use this *.d.ts with our editor and the typescript compiler, we have to add it to our tsconfig.json, for example adding the whole ./src/protobuf folder to the typeRoots array.

Now we can start using protobuf by importing the bundle

import * as root from '../protobuf/value.protobuf.js';

As defined in value.proto, the object under root is the packagename example and then the only message type we defined Value. With it we can now create, encode, decode, verify and convert messages. For detailed information, what you can do with the Value object, you may hava a look at the typescript definition file ./src/protobuf/value.protobuf.d.ts

If you run the example app, click the “Subscribe” button, and enter a number. Remember, we defined the value as a uint64. Then click “Publish”.

For this example, I chose the catchy topic name angular2-protobuf-mqtt-example. In a real application this wouldn’t be a clever idea, since the topic would likely make up the majority of the message size. So keep in mind, to chose concise topic names.

Now the following code is executed in the publish component

const buffer = root.example.Value.encode({value: 123456789}).finish();
this.mqtt.unsafePublish('angular2-protobuf-mqtt-example', buffer);

The subscribe component now will receive the encoded message and will decode it

const value: root.example.Value = root.example.Value.decode(message.payload);
this.message = value.toJSON();

this.message is then rendered in the component via the json pipe.