Category: NoSQL

Overview of MongoDB’s BaaS offering: MongoDB Stitch

One of the challenges many developers have is keeping up with backend processes. Things like data security, service integrations, and code for data manipulations can take up a lot of time and resources. As a result, many have made a move to utilize a Backend as a Service (BaaS). If you have an application backed by MongoDB there is a new Baas option available, MongoDB Stitch.

MongoDB allows you to get started at no cost with Stitch, which is always great. At the moment it is backed by MongoDB’s Database as a Service (DBaaS) offering, Atlas. In another post, I discussed Atlas and am excited that these two services are connected.

BaaS Overview

A Backend as a Service approach to development allows mobile and web developers to connect their applications to backend cloud storage and utilize inherent processing capabilities. Further, it provides many common features that users demand from sites and applications. Some of these features are security, user management, push notifications, and social networking integrations.

MongoDB Stitch Benefits

Stitch provides many capabilities for developers to leverage the features inherent in MongoDB along with BaaS concepts. It provides direct access to the database, whether it is already existing or new data, Stitch allows developers to focus on building their application. The backend logic is taken care of and provided. This leads to a faster development cycle.

Stitch takes MongoDB’s data security features even further. It provides functionality such as end user authentication and access control on a per-field basis. For example, if the marketing department doesn’t need access to a customer’s financial information, those fields can be excluded from their access rights. This allows developers and application stakeholders to have confidence that information and services are only available to the proper users.

MongoDB has built Stitch in an open fashion. It provides a single API allowing access to both the MongoDB database as well as other public cloud services. This allows for microservice integration and prevents being committed to a single vendor.

Another terrific benefit is the integration with Atlas. This allows for easy scaling of capacity and performance as an application grows. This is yet another set of duties that don’t need to be managed ourselves. We can allow the details of the backend application and infrastructure to be handled by a team of Dev-Ops folks at MongoDB and can then concentrate on the application itself.

MongoDB Stitch Components

There are three main components of Stitch, pipelines, services, and rules. Services come in two different varieties, MongoDB services, and partner services for integration with services like Amazon S3 or Twilio for messaging. Let’s take a quick look at what these different components do and how they fit into an application.

Services

Services in Stitch allow for application integration and execution of common actions. There are integrations for a variety of third-party (partner) services for such tasks as user authentication through Facebook or Google, Slack, Twilio, and some components of AWS, to name a few. The HTTP Service allows for a MongoDB Stitch application to connect to REST API services. MongoDB has a dedicated service for connecting to Atlas and has some built in pipeline operations.

Pipelines

A Stitch pipeline allows for a sequence of actions to be performed in order by the Stitch services. This is a powerful feature. It allows a series of actions to be defined all through simple JSON syntax.

A pipeline is built in stages, each running in consecutive order. A stage runs and passes the information it generates onto the next stage. For example, you can do a MongoDB aggregation stage from your data and determine which user in the past week had the most likes on your new Snap-o-gram app. That stage could then pass that information to the Twilio Service which could send a text to their phone with this awesome accomplishment.

There are some Built-In Actions for filtering the pipeline input, defining in an explicit way what the output of a stage will be, expression evaluation against input documents, and a project feature, to name a few. Another nice feature MongoDB has provided is the concept of named pipelines. This allows for a designed pipeline to be reused within a Stitch app and referenced by name. A tremendous help for writing code that follows the DRY (don’t repeat yourself) principle.

Rules

The last major component of a MongoDB Stitch is Rules. Rules, as one might guess, allow for a control over the actions a service takes. Rules are designed and written in JSON format, as are pipelines.

One can define a rule for read, write, and validation operations, for example. These can be used at the document level and on down to the field level. Want to prevent a service from reading specific financial data in a document? Write a rule for that. It is a powerful feature of MongoDB Stitch and provides an extra level of security for your data.

MongoDB Stitch Development

Right now there are three different options for developing a Stitch application. Using JavaScript for web applications and for mobile development there are options for Android and iOS. MongoDB has put together some great tutorials and getting started guides.

Wrap Up

The name Stitch comes out of the idea of stitching together the pieces of an application and not from the Disney character in Lilo & Stitch. Given the benefits and features MongoDB Stitch brings to the table, I think it is a very appropriate name. Much like sewing brings all of the pieces of a garment together, MongoDB Stitch does the same thing for your application.

I know that for myself I plan on utilizing this service on my next project.

As with any “pre-packaged” service, one gives up some flexibility and control over your application. However, for being able to retain direct access to your MongoDB database, and the collections and documents it contains, Stitch is a great option. Keep in mind that as of this post, Stitch is still in a beta version. As with any beta product, things may change with the final product.

There are several MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB for the definition of authentication?” and get a helpful response.

Follow me on Twitter @kenwalger to get the latest updates on my postings.

Facebooktwitterredditlinkedinmail

Importing data with mongoimport

There comes a time in almost everyone’s experience with databases when it would be great to bring in data from an outside source. Often the data is in a spreadsheet format (CSV or TSV) or perhaps a JSON format. I discussed some of the command line tools MongoDB provides in a previous post. Importing data into a MongoDB database is made easy with the CLI tool, mongoimport.

For many use cases, mongoimport is pretty straight forward. It is, in fact, highly used in the MongoDB University courses as a way to quickly populate a database, for example. I’d like to look at some use cases beyond simply populating an empty collection, however.

mongoimport

Connections

The mongoimport will connect to a running mongod or mongos, instance running, by default, on port 27017 on localhost. The syntax of the mongoimport command is fairly straightforward. If for example, we want to populate the posts collection in the blog database with a posts.json file it is simple enough to run the following command.

mongoimport --db blog --collection posts --file posts.json

That is pretty easy. We can make it easier too by using the shorthand version of those flags.

mongoimport -d blog -c posts --file posts.json

If we want to make sure that our posts collection is dropped and only the new data is there, we can use the --drop flag.

mongoimport -d blog -c posts --drop --file posts.json

If you need to change the host or port number, there are flags for that as well, --host and --port, respectively. --host is even more convenient because it allows you to add the port at the end, and use a shorter flag -h. So the following are the same:

mongoimport --host 123.123.123.1 --port 1234 -d blog -c posts --file posts.json

mongoimport -h 123.123.123.1:1234 -d blog -c posts --file posts.json

That’s easy enough as well. What if, however, our MongoDB server requires user authentication, like any good server should?

mongoimport Server Authentication

Data security should be on everyone’s mind when it comes to server management. With that in mind, MongoDB offers a variety of ways to secure your data. Assuming that one needs to get authenticated access to the server, how can one use mongoimport to do so? You guessed it, there are flags for that too. --username or -u and --password or -p are your friends.

mongoimport -h 123.123.123.1:1234 -u user -p "pass" -d blog -c posts --file posts.json

We can add in some extra assurances by leaving off the --password flag and mongoimport will prompt for an appropriate password.

That works great for some simpler authentication options, but what if we have a more involved authentication system with an authentication database? We can specify one with the --authenticationDatabase flag. That’s pretty handy to keep only authorized people from importing data into your collection.

mongoimport provides a great range of flag options for connecting to secured servers. I would highly recommend looking at the documentation for specifics based on your environment.

File & Column Types

As stated earlier, mongoimport works on CSV, TSV, and JSON documents. By default the import format is JSON. With the --type flag we can import CSV or TSV files. Since CSV and TSV files can contain some special features, let’s look at some of the options for working with them and mongoimport.

Many times a CSV or TSV file will include a header line. It would be handy if we could utilize those header values as field names in our MongoDB documents, right? Well, mongoimport offers a --headerline flag that accomplishes that for us.

For times in which our CSV or TSV file doesn’t include header information, mongoimport has a solution for that as well. With the --fields flag, one can provide a comma-separated list of field names. Alternatively, you can generate a file of field names, with one name per line, and pass it along with the --fieldFile flag.

Along with some of the other features new to MongoDB version 3.4, there are some new features added to mongoimport. One of them is the
--columnsHaveTypes flag. When used in conjunction with the --fields,  --fieldFile, or --headerline flag it allows you to specify the types of each field. You pass in the field name in the format of columnName.type() along with any arguments into the type() method. So, for example, if you were typing a filed called isAdmin you would use isAdmin.bool(). Have a look at the --columnsHaveTypes documentation for a list of available types and supported arguments.

Fair warning here, the flags dealing with header information are for CSV and/or TSV files. If one attempts to use them with a JSON formatted file, mongoimport gets grumpy and returns an error.

Importing into an existing collection

One last concept and list of flags I’d like to cover is for those instances in which you want to import data into an existing collection. The
--mode flag offers a way to tell mongoimport how to handle existing collection documents which match incoming ones. There are three options to the --mode flag, insert, upsert, and merge.

  • Insert allows the documents to get put into the collection with the only check being on fields with a unique index. If there are duplicate values, mongoimport logs an error.
  • Upsert replaces documents in the database with the new documents from the import file. All other documents get inserted.
  • Merge, well, it merges existing documents with matching incoming documents and inserts the others. This is another new feature of version 3.4.

If you are needing to import documents in one of these ways, look at the documentation for options on upserting and merging based on field other than _id.

Wrap Up

MongoDB also provides a similar, but inverse, function mongoexport. While both tools are powerful they do not preseve the BSON data types than MongoDB uses. As such, these tools should not be used for production backups. MongoDB provides other tools for backup methods.

I hope that this post has given you some insights into one of the powerful MongoDB Package Component tools that are provided “out of the box”, mongoimport. Some programming languages have developed their own separate tools for importing data. Some of them are better than others. For me, since such a powerful import tool is already provided, I find myself using mongoimport more often than not.

If you haven’t tried it out yet yourself, I would encourage you to do so.

There are several MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB for the definition of authentication?” and get a helpful response.

Follow me on Twitter @kenwalger to get the latest updates on my postings.

Facebooktwitterredditlinkedinmail