Change Streams in MongoDB 3.6

Gone are the days when expectations of information were not instantaneous access to the data. In today’s world, people expect real-time results and information. What’s the moisture content of the soil around grapevine X right now? Not yesterday, not five minutes ago, but now. Being notified about constantly changing data points is critical in many of today’s applications. In the latest 3.6 release of MongoDB a new feature is being introduced to assist with these real-time expectations: Change Streams.

Change Streams Defined

Change streams are being implemented in the driver with a new aggregation operator, $changeStream and watch method in the API. We can now specify a $changeStream stage at the beginning of our pipeline and request that notifications are sent for specific changes to a particular collection.

Example Use Case

Let’s take the soil moisture example I stated earlier. Imagine that you have a farm with multiple moisture sensors or a TMP36 Analog temperature sensor connected to something like a NodeMCU ESP8266 or LoPy device. Without going into detail about how to set the IoT network up, for the sake of our use case we can assume that they are sending readings back to a MongoDB server. Perhaps the data is being stored on MongoDB Atlas even. I have, however, written about the LoPy microcontroller and ESP8266 with temperature sensors in the past if you are interested in learning more.

Anyway, for the sake of argument, let’s say that we need to know when the moisture content of the soil drops below a certain level. Moisture values on similar devices are measured as a raw number between 0-1024. The higher the number, the more moisture in the soil. For our use case, we need to know when the moisture level drops below 350 (too dry).

Change Stream in action

Since these recordings are frequently changing, we can set up a change stream to get notified. It’s fairly straightforward to establish a notification with the watch method. We could set something like this up:

change_stream = client.winery.collection.watch([
  {'$match': {
       'operationType': {'$in': ['insert']}
    }},
    {'$match': {
        'moisture_level': {'$lt': 350}
    }}
]);
for change in change_stream:
    turn_on_water();
    # Something else magical

Now when we have an insert into the database where the moisture_level is below 350, we’ll turn our watering system on along with whatever else needs to be done. We could set up a similar change stream for when the moisture level gets high enough to warrant shutting the system off at 450.

This is but one example of where change streams would be a useful tool. Updating reports dynamically on data changes, publish/subscribe types of functionality, brokers, and many other use cases are now open to native MongoDB actions.

Change Steam Details

You’ll notice that there is an operationType being looked at there. Change streams look at five different events, or Operation Types

  • Insert
  • Delete
  • Replace (everything but the unique id)
  • Update
  • Invalidate (in cases of an invalid cursor being returned)

Change streams also utilize access controls and can be set up based on user roles. They present a defined API and enable scaling across primaries and secondaries. MongoDB is supporting up to 1,000 concurrent change streams per data bearing node in 3.6.

They are durable, meaning they rely on the data being majority committed in a replica set. The order of the changes in data is important too, right? 3.6 also introduces a global logical clock which, among other uses, will ensure that clients always will get changes in the order they occurred on the server. This is very important for replace and update operations.

Wrap Up

There are a lot more technical details about change streams forthcoming as 3.6 moves closer to production and documentation is further flushed out. Change streams are inherent in the 3.6 Server and are available in both the community and enterprise versions of the product. MongoDB Version 3.6 is available to download today as a release candidate. Which means that while the high-level concepts of change streams discussed here likely will remain in tack in the final release, there may be some subtle changes. That being said, change streams are a great feature being implemented in the new release. If your applications require or could use real-time information, I would encourage you to have a look.


Follow me on Twitter @kenwalger to get the latest updates on my postings.

There are a few MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB for the definition of a document?” and get a helpful response.

Facebooktwitterredditlinkedinmail

A review of the LoPy microcontroller

In an effort to stay current with microcontrollers, I recently picked a new MicroPython enabled device, the LoPy. Like the ever popular NodeMCU ESP8266, it is capable of utilizing many different components and gadgets. However, to LoPy comes with MicroPython installed so flashing the device isn’t necessary to write projects in Python.

The LoPy board is produced by PyCom which is based out of the United Kingdom and was released towards the end of 2016. It is more powerful and feature-rich than the NodeMCU which drives the price up over the ESP8266 boards. It also requires the use of an expansion board to build projects. Ok, requires might not be entirely accurate, strongly recommended might be better. Fortunately, the expansion board can be used with PyCom’s other products as well.

LoPy Specs

The LoPy comes with 512kB of RAM, versus 160kB for the ESP8266. It can handle up to four times as much flash memory (4MB) as well and uses the Espressif ESP32 chipset. All of that opens things up for larger IoT projects. One really cool and interesting feature with the LoPy is the addition of a LoRa radio frequency system in addition to the other radio system

This little board has three different radio systems on it.

  • LoRa
  • WiFi (802.11 b/g/n 16Mbps)
  • Bluetooth, both Bluetooth Classic and Bluetooth Low Energy (BLE)

The device accepts, and in fact, in the case of LoRa requires the use of external antennas, there are connectors for one WiFi antenna and one LoRa antenna. The use of the external antennas yields some impressive distances for operating ranges. For WiFi the range is a reported 1km and for LoRa, in perfect conditions as a network node the range is specified as 40km.

PyCom LoPy Device

It provides support for a variety of security and hash/encryption methods such as TLS/SSL. For power, it takes 3.3V – 5.5V of input and with the expansion board easily connects to power with a USB connection. Additionally, it also allows for a battery to be connected for powering the device away from a USB connected computer.

LoRa

I am going to assume that most people are familiar with WiFi and Bluetooth technologies in general terms. I will also assume that most people are not familiar with LoRa, so let’s spend a little time discussing it.

At a high level, LoRa is designed as part of a moderate range wireless network called Low Power Radio Wide Area Network (LoRaWAN). The focus of LoRaWAN is to provide IoT devices a standard and seamless communication system to communicate between IoT devices covering large areas. In a LoRaWAN network data is sent digitally from one node to a neighboring node. If that node is not the data’s final destination, it is forwarded on until the final destination is reached. This final destination is often a gateway connection to the Internet allowing for global reach.

Most of us in the developed world are used to very high speeds when it comes to network access. For example, I just tested my internet speed and had just over 48 Mbps for download speed at 6 Mbps for upload. LoRaWAN can only dream of these speeds. LoRaWAN tends to max out around 50 Kbps, yes, that’s a K there. Under 0.5 Kbps is common on the slow end. This may send chills down your leg at first glance, but really, what types of data are typically sent with an IoT device? Most applications are sending basic sensor data and not streaming video or audio.

LoPy LoRa settings

As most of us are painfully aware, there are global forces at work when it comes to management of things like radio waves. The same holds true for the frequencies on which LoRa operates.

Region Frequency in MHz
North and South America, Australia and New Zealand 915 MHz
Europe 868 MHz

You will need to configure the LoPy to utilize the proper frequency. This is accomplished through updating the firmware (a good practice anyway) with a selection of the country. Fortunately, these frequencies and devices fall under the category of low-power transmitters in the ISM band and don’t need to be licensed.

Wrap Up

The features, power requirements, and radio capabilities make the LoPy a very interesting IoT device. Especially for those that want to be able to utilize a flavor of Python to build their projects. In fact, the LoPy would be a great candidate for nodes in a mesh network. I’ll talk about the idea of a mesh network in a future post.

I hope that this brief overview of the LoPy has shown some interesting features. Perhaps it will expand the possibilities for your IoT projects. The capability to use a LoRaWAN network to cover long distance communications is particularly intriguing.


Follow me on Twitter @kenwalger to get the latest updates on my postings on MicroPython and IoT and let me know what you are building with MicroPython.

Facebooktwitterredditlinkedinmail