Schema Validation in MongoDB 3.6

MongoDB 3.6 brings lots of great new features with the new release. I’ve already covered Change Streams and Retryable Writes in previous posts. This post will cover a feature which expands upon the document validation feature from MongoDB 3.2, schema validation. Schema validation allows for teams to define a prescribed document structure for each collection.

There are times when enforcing strict data structures and content are required, even with the flexible schema and document data model that MongoDB provides. Schema validation allows for the ability to define a prescribed document structure for each collection. If one tries to insert or update a document which does not conform to the applied structure the operation can be rejected. The rules for document structure are based on the JSON schema draft specification.

Schema Validation

Let’s take a quick look at how this works in action before diving into a discussion of feature benefits.

Imagine a collection of food recipes. Each recipe will have a recipe name, number of servings, cooking method, ingredients, and list of instructions. For the ingredients, we want to enforce that there is a numerical quantity, a measure of quantity, the ingredient name, and an optional value for any prep work on the ingredient, such as “peeled” or “brunoise“.  For our example here my options are not all-inclusive for options but simply serve as examples. I hope my former culinary colleagues forgive me.

We’ll begin with creating a recipes collection and assigning the schema validation rules with the validator option and the new $jsonSchema operator.

db.createCollection( "recipes",
{
  validator: 
    {
      $jsonSchema:
        {
          bsonType: "object",
          required: ["name", "servings", "ingredients"],
          additionalProperties: false,
          properties:
            {
              _id: {},
              name: {
                bsonType: "string",
                description: "'name' is required and is a string"
                    },
              servings: {
                bsonType: ["int", "double"],
                minimum: 0,
                description: "'servings' is required and must be an integer greater than zero."
                    },
              cooking_method: {
                 enum: ["broil", "grill", "roast", "bake", "saute", "pan-fry", "deep-fry", "poach", "simmer", "boil", "steam", "braise", "stew"],
              description: "'cooking_method' is optional but, if used, must be one of the listed options."
                    },
              ingredients: 
              {
                bsonType: ["array"],
                minItems: 1,
                maxItems: 50,
                items: {
                  bsonType: ["object"],     
                  required: ["quantity", "measure", "ingredient"],
                  additionalProperties: false,
                  description: "'ingredients' must contain the stated fields.",
                  properties: 
                  {
                    quantity: {
                      bsonType: ["double", "decimal"],
                      description: "'quantity' is required and is of double or decimal type"
                            },
                    measure: {
                      enum: ["tsp", "Tbsp", "cup", "ounce", "pound",  "each"],
                      description: "'measure' is required and can only be one of the given enum values"
                            },
                    ingredient: {
                      bsonType: "string",
                      description: "'ingredient' is required and is a string"
                            },
                    format: {
                      bsonType: "string",
                      description: "'format' is an optional field of type string"
                            }
                  }
              }
           }
        }
     }
   }
})

Our validator can include many more rules that are beyond the scope of this introduction to schema validation. However, we can see how powerful this feature is in this small example. Let’s look at some example documents to insert as well to see what would happen if we try to insert a document into the collection.

Sample Inserts
db.recipes.insertOne({name: "Chocolate Sponge Cake Filling", 
servings: 4, 
ingredients: [{quantity: 7, measure: "ounce", ingredient: "bittersweet chocolate", format: "chopped"}, 
{quantity: 2, measure: "cup", ingredient: "heavy cream"}
]})

This insert works since it covers all of the required fields in their proper format. It also doesn’t include any invalid extra fields which would be prohibited since we have additionalProperties: false set. If we were to try to insert the following document, however, we would get an error.

db.recipes.insertOne({name: "Chocolate Sponge Cake Filling", 
servings: 4, 
ingredients: [{quantity: 7, measure: "ounce", ingredient: "bittersweet chocolate", format: "chopped"}, 
{quantity: 2, measure: "cup", ingredient: "heavy cream"}],
directions: "Boil cream and pour over chocolate. Stir until chocolate is melted."
})

Since we added in a directions field into our recipe document, the insert will fail with additionalProperties: false set. I should add in an optional directions field into the schema validation to allow for that as directions are indeed important.

Schema Validation Benefits

Even with these basic examples, I think it is clear that schema validation is a great new enhancement. The flexibility of a dynamic schema in MongoDB can now easily be paired with data governance controls over an entire collection. While there are lots of practical benefits from this, here are some specific benefits.

  1. Application logic simplification. With a strictly defined definition of what a collection looks like, there isn’t a need for the application to handle the guarantees and associated errors.
  2. Control of data. Client applications must adhere to the rules set forth by the collection. No need to worry about data being updated or inserted that has incorrect field names or data types.
  3. Governmental compliance. There are applications and data models in a variety of industries and locales that require data to be stored in specific formats. For example, the EU General Data Protection Regulation in the European Union.

Wrap Up

With MongoDB 3.6 schema validation, administrators have tunable controls over the database. Documents that don’t conform to a prescribed set of conditions can be rejected or still written and a message logged about the action. It also brings with it the ability to query based on the schema definition to allow, for example, a search on all documents that don’t conform to the schema.

Schema validation is another exciting feature in the 3.6 release of MongoDB. I would urge you to download it and try it out. I’d bet you’ll be as excited about it as I am.


Follow me on Twitter @kenwalger to get the latest updates on my postings. Or better yet, sign up for the email list to get updates in your mailbox!

There are a few MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB for the definition of a document?” and get a helpful response. I also created a culinary skill for the Echo if you’d like to update your culinary knowledge as well.

Facebooktwitterredditlinkedinmail

MongoDB Stitch Tutorial

I have talked about MongoDB‘s Backend as a Service (BaaS) Stitch previously. In this post, let’s take a look at a basic Stitch application and how easy it is to get started. For this particular tutorial, we’ll be looking at basic Stitch functionality. However, I’ll point out some options along the way for extending the application.

The Stitch Application Overview

For this tutorial, let’s imagine an application that tracks blood glucose for diabetic patients. A blood sugar monitor, like the One Drop Chrome, would send results to our application. Medical practitioners could see the results and leave comments on the reading. While beyond the scope of this tutorial, Stitch can integrate with a service such as Twilio to send the comments via SMS message to the patient for diabetic management suggestions. From a development perspective, this can be integrated with MongoDB 3.6 Change Streams as well.

A sample application page is available to view here with working comments attached to a MongoDB Atlas cluster and using Stitch. Let’s see how it was generated.

Cloud Services

Stitch runs on top of Atlas so we’ll need to set those services up.

Atlas

First off we’ll need to log into MongoDB Atlas or create an account. Then we’ll need to set up an Atlas Cluster. There is an option for an M0 level which is free and doesn’t require a credit card to get started. This is a great feature to be able to try out this incredible service.

Stitch is currently only located in the AWS US East 1 region so it is best to set your cluster up in that region for optimal performance. Once the cluster is configured, deployed, and initialized we’ll move onto setting up Stitch.

Stitch

To get started setting up a Stitch application, in the MongoDB Atlas console, select the Stitch App option.

Stitch Application Selection

Then the Create New Application button in the upper right. Give the application a name, I’m calling this application glucose, and select the MongoDB Atlas Cluster you want to connect to.

New Stitch App

It will take a minute to spin up the new application. Once it is up and running, we’ll want to set authentication to Anonymous Authentication.

Anonymous Authentication

There are a few pieces of information we’ll need from this screen for our application. For this tutorial, we’ll be using the information from JS(Browser) but it is great to see support for Node.js, iOS, and Android configurations as well.

Stitch client setting strings

We’ll want to grab the <script> information from the Importing on a Webpage bit and add that to our HTML that we’ll generate. We’ll also need the App Id listed later on as well. But first, let’s setup our database and collection. We’ll want to click mongodb-atlas from the left navigation panel and then the Rules tab.

Connect the Database

Click NEW to add a new MongoDB collection. For the database, I’ve called mine glucose and I called the collection results. Now we’ll need to configure some permissions to allow, for the purposes of this tutorial, anyone to read the comments. Obviously, with sensitive and private medical data we would want to set the permissions to be more strict.

Under the Rules tab, click on the collection you just created and proceed to the Filters tab.

Delete this existing filter and click SAVE. Now head on over to the Field Rules tab. In the Permissions for top-level document section, we’ll want to change the Read rule to {} and click SAVE.

With these settings, we are allowing anyone to read any records in glucose.results, but only the owners can edit or delete their own comments. Stitch has integration with services such as Facebook and Google via OAuth. Further, custom authentication is possible with JSON Web Tokens.

Stitch the Application to the HTML

Now that we’ve set up our Stitch application on the backend, we need to integrate it into our HTML page. For this tutorial, we’re generating a static page with fake data. However, in a real application, we would use some dynamic content. At any rate, let’s see how we can proceed.

HTML

We’ll generate a basic HTML page with our <script> information in the <head> section. And we’ll include a form at the end to be able to get input from our medical professionals.

<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="x-ua-compatible" content="ie-edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>Blood Glucose</title>
        <script src="https://s3.amazonaws.com/stitch-sdks/js/library/stable/stitch.min.js"></script>
    </head>
    <body>
        <h2>Sample Blood Glucose Report</h2>
        <div>
            <p>Patient Id: 12345</p>
            <p>Date: 5 November 2017</p>
            <p>Time: 07:23am</p>
            <p>Reading was taken: Before Breakfast</p>
            <p>Result: 110</p>
            <hr>
            <div id="comments">
            </div>
            <hr>
            <form>
                <label for="new_comment">Add Comments:</label>
                <input id="new_comment">
                <input class="button" type="submit">
            </form>
        </div>
    </body>
</html>

Let’s also think about what we’re trying to do with our JavaScript and add in the appropriate hooks in our HTML before we tackle our JavaScript.

Since we want our comments to load when the page does, let’s add an onload to the <body> tag:

<body onload="displayCommentsOnLoad()">

We’ll also need to add an on click listener to our form:

<input class="button" type="submit" onClick="addComment()">

Nice. It seems like, then, that we have three JavaScript functions to write to get our functionality implemented. The two mentioned and a third displayComments seems like it will be necessary. Let’s tackle those functions next in glucose.js.

JavaScript

glucose.js

const client = new stitch.StitchClient('glucose-XXXXX');
const db = client.service('mongodb', 'mongodb-atlas').db('glucose');

function displayCommentsOnLoad() {
    client.login().then(displayComments();
}

function displayComments() {
    db.collection('results').find({}).then(docs => {
        const html = docs.map(c => "<div>" + c.comment + "</div>").join("");
        document.getElementByID("comments").innerHTML = html;
    });
}

function addComment() {
    const foo = document.getElementById("new_comment");
    db.collection("results").insert({owner_id : client.authedId(), comment : foo.value}).then(displayComments);
    foo.value = "";
}

You’ll want to alter the glucose-XXXXX to utilize the App Id for your own application and if you are using a different database and/or collection names, make those changes as well.

We’ll also need to add the script to our HTML in the <head> section:

<script src="glucose.js"></script>

I have the completed project code, with some minimal Foundation styling available on GitHub as well.

Wrap Up

And that’s it! Neat. We now have a basic web page that will accept comments from a user (medical professional) and, through the power of Stitch, save the comments in MongoDB Atlas hosted database. Stitch has provided the backend power and we just had to write some minimal JavaScript functions to handle the button click and document insert into our collection.


Follow me on Twitter @kenwalger to get the latest updates on my postings. Or better yet, sign up for the email list to get updates in your mailbox!

There are a few MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB for the definition of a document?” and get a helpful response.

Facebooktwitterredditlinkedinmail