MQTT & Telegraf data to UltraTimeSeries

This article will show how to set up the MQTT UltraTimeSeries output plugin from Telegraf, this way MQTT data can go into UltraTimeSeries in a very simple way.

MQTT

MQTT Broker + Data Setup

You’ll need an MQTT broker so you can publish and subscribe to data. In this case, we’ll use Eclipse Mosquitto.

You’ll also need an MQTT data generator, we’ll be using ufthelp’s MQTT Generator.

Once you have downloaded/installed both these tools, you can try them as follows, if you’d like:

  • On Eclipse Mosquitto:

On the terminal, subscribe to topic sensor/70ff2d82-972e-11e6-8bde-4485001bc64b/data/test with the command:

mosquitto_sub -t sensor/70ff2d82-972e-11e6-8bde-4485001bc64b/data/test
  • On the MQTT data generator folder:

Configure the config.json file as desired, for example:

{
"mqtt": {
"host": "localhost",
"port": 1883,
"topic": "sensor/70ff2d82-972e-11e6-8bde-4485001bc64b/data/test"
},
"sensors": {
"Sensor 1": {
"c1": 1,"c2": 12
},
"Sensor 2": {
"c1": 2,"c2": 14
},
"Sensor 3": {
"c1": 3,"c2": 33
},
"Sensor 4": {
"c1": 4,"c2": 34
},
"Sensor 5": {
"c1": 5,"c2": 22
}
}
}

On the terminal, write python3 mqttgen.py NoOfMessages Interval(ms) ThreadCount, for example:

python3 mqttgen.py 100 10 1

and you’ll see your subscription to the topic working:

MQTT Subscription

UltraTimeSeries instance setup

For this demonstration, you’ll need a UltraTimeSeries instance running. This can be achieved very easily with docker, have a look at How To Get Started With UltraTimeSeries Using Docker.

InfluxDB and Telegraf Setup

To set up these two tools, please have a look at InfluxDB and Telegraf.

MQTT UltraTimeSeries Plugin Setup

After this, it’s time to start working on the Telegraf folder. Under telegraf/plugins/outputs, let’s create a folder for the mqtt-uts plugin. Create a Go file inside the folder we just created, let’s call it mqtt-uts.go.

For example, you can use this configuration file:

uts.go

It is necessary to add the plugin on the all.go file under telegraf/plugins/outputs/all; just add this line on the document:

"github.com/influxdata/telegraf/plugins/outputs/mqtt-uts"

On the Influxdb instance, create a bucket (adv-mqtt) and enter the Telegraf configuration that’s on the mqtt-uts.go file, which is the sampleConfig variable.

  • On the telegraf/ folder:
  1. Type make
  2. Go to your buckets page, then click on that bucket’s Setup Instructions, then on step 2: Generate New API Token and Copy to Clipboard. Paste it on the terminal.
  3. Still on Setup Instructions, on step 3, Copy to Clipboard the Telegraf start command and paste it on the terminal.

Telegraf Setup Instructions

Now it should be good to go, just try to publish some data on the desired topic and go to the UltraTimeSeries instance and check if it appears there.

UltraTimeSeries Action Details

Generating big data samples

To generate bigger datasets than the one from MQTT Generator, have a look at the following Python Script:

mqtt-uts-gen.py

You can configure the script as you’d like.

For example, if you wish to modify the Dimension and Unit of the time series, you’ll find that option on mqtt-uts.go file, under Measure (Dimension and Unit):

Measure: Dimension and Unit

You can find all units under UltraTimeSeries Postman collection, under Messages → Get All Units endpoint.

After that, just run it to create a sample-for-mqtt-gen.json file with the necessary configuration: python3 /home/pat/Documents/est/twoimpulse/uts/mqtt-gen.py.

The resulting .json is run by changing line 44 of mqttgen.py, the config_path variable value should be replaced with the value sample-for-mqtt-gen.json.

Then, just run the mqttgen.py file as expected. This will publish data on the “ovens” topic and will be accessible on the UltraTimeSeries instance, you just need to configure some parts of the MQTT Telegraf plugin, both on the mqtt-uts.go file and on the Influxdb instance plugin configuration: json_string_fields and topics (in this case, ["Temperature 1", "Temperature 2", "Temperature 3"] and ["ovens"], respectively). Then, restart Telegraf, and it’s good to go.

Example of the generated file:

sample-for-mqtt-gen.json