MongoDB CRUD with Python

In my last post, we looked at how to combat Impostor Syndrome and came to the conclusion that deepening our knowledge on a given topic helps to sharpen of knowledge sword to fight our feelings of self doubt. If you missed that post, take a look here. This week let’s take a look at using the NoSQL database MongoDB with Python. In a related post, I’ll show how to implement them into web frameworks.

Planning

We will start with an exploration of CRUD operations in MongoDB using Python and PyMongo. Next, we’ll have a look at the Python web framework Bottle and then use Bottle with MongoDB to generate a simple web site. Then we’ll spend a bit of time using MongoDB with the Django web framework to generate a slightly more involved site than our Bottle version.

In this post, we kick off this series with a simple discussion of how to do some simple CRUD operations in Python similar to my post for doing CRUD operations in Java. Not familiar with CRUD? It stands for Create, Read, Update, and Delete. Those are the basic operations one will find themselves doing with any database, not just MongoDB.

In December of 2016, MongoDB released a new version of their database, 3.4, and I’m using 3.4.1 for this series of posts. While we aren’t going to go into any release specific features this week, we might in future posts, so I would encourage you to utilize the latest version. We will be touching on some of the features in the latest version of Python, 3.6, later on as well so make sure you are utilizing that version as well.

Python Driver – PyMongo

Okay, let’s concentrate on some basic implementations and general syntax of the Python driver for MongoDB as well as some basic database documents. Documents, by the way, are similar to a SQL database record.

The project code for this post will be available on GitHub and a link will be included at the end of the post. Let’s start off with our requirements.txt file to list out our project dependencies:

pymongo=='3.4.0'

With that in place we can do pip install -r requirements.txt to install the required version of the libraries for our project.

Another thing we will need for our project to work is a running instance of a MongoDB server. I am using version 3.4.1 and will assume that you have MongoDB 3.4 downloaded, installed, and running on the default address of localhost and port 27017. Further, since we are just sharpening our sword of knowledge here and don’t need MongoDB server authentication, let’s assume that feature is turned off or not implemented at all on our server.

Great, now that our environment is setup let’s enter into the planning stages a bit. What do we need to do?

  • Connect to the database
  • Insert an individual document into the database (Create)
  • Find and print a document in the database (Read)
  • Update the document in the database
  • Finally, Delete the document from the database

Let’s get started!

Database Connection

To setup our connection we will need to import MongoClient from pymongo, set the host and port we intend on using, and then select the database and collection we need. Remember that in MongoDB a collection is similar to a table in the SQL world in that it is a namespace for our data.

from pymongo import MongoClient

HOST = 'localhost'
PORT = 27017

client = MongoClient(HOST, PORT)
database = client.cookbook
recipes = database.recipes

Great, we now have a connection to the MongoDB database cookbook and the collection recipes. But wait, we didn’t ask to specifically create either of those, nor did we put anything into the collection yet. That’s correct, MongoDB generates that for us and waits for us to start using cookbook.recipes. Now, we need to add a basic recipe to our collection so that we have something with which to work.

We’ll define a single recipe document and insert it into our collection using the insert_one() method. Here‘s a link to further documentation on PyMongo’s collection level operations.

recipe = {'title': 'chocolate milk',
          'description': 'Yummy drink',
          'ingredients': [
              {'name': 'milk', 'quantity': 8, 'unit of measure': 'ounce'},
              {'name': 'chocolate syrup', 'quantity': 2, 'unit of measure': 'ounce'}
          ],
          'yield': {'quantity': 1, 'unit': 'glass'},
          'prep time': 0,
          'cook time': 0,
          'author': 'Biff Tannen',
          'uploaded_by': 'kenwalger',
          }

recipes.insert_one(recipe)

That’s all there is to it. We have just added a Chocolate Milk recipe to our collection. We just call the insert_one() method on our collection and pass in our document as an argument.

Reading the documents

Getting a document back out is pretty straight forward as well, we can simply use the find_one() function, which is a collection level operator from the PyMongo driver. It will, as the name would indicate, find a single document in the collection. This can be very useful to get an idea about the schema of the collection. Since we currently only have one record in our database, it should be perfect for getting our information out. I’m also going to include a new import pprint so that when we print out our data in a pretty fashion.

import pprint

print("\nPretty Printed document: \n")
pprint.pprint(recipes.find_one())

Our output should be as follows, with the _id value being different on your system:

Pretty Print: 

{'_id': ObjectId('588541a0146bde28a08217d4'),
 'author': 'Biff Tannen',
 'cook time': 0,
 'description': 'Yummy drink',
 'ingredients': [{'name': 'milk', 'quantity': 8, 'unit of measure': 'ounce'},
                 {'name': 'chocolate syrup',
                  'quantity': 2,
                  'unit of measure': 'ounce'}],
 'prep time': 0,
 'title': 'chocolate milk',
 'uploaded_by': 'kenwalger',
 'yield': {'quantity': 1, 'unit': 'glass'}}

Cool! We have seen how to do two of the four CRUD tasks already! And through the pymongo driver, it has been relatively straightforward to do so. We have a basic recipe for chocolate milk in our cookbook.

Update and Delete

But wait, we incorrectly attributed the recipe to Biff Tannen. As any fan of the Back to the Future movie series knows, chocolate milk is George McFly’s drink, not his arch enemy Biff’s. Let’s Update our database with that change using the update_one() method with the $set operator and print out the new record to make sure it accomplished what we wanted.

recipes.update_one({'title': 'chocolate milk'},
                   {'$set': {'author': 'George McFly'}
                    }
                   )
print("\nShould be George McFly: ")
pprint.pprint(recipes.find_one({'author': 'George McFly'}))

Great, and our output should now reflect that change and we can tell it is the same record because the collection unique _id value is still the same.

Should be George McFly: 
{'_id': ObjectId('588541a0146bde28a08217d4'),
 'author': 'George McFly',
 'cook time': 0,
 'description': 'Yummy drink',
 'ingredients': [{'name': 'milk', 'quantity': 8, 'unit of measure': 'ounce'},
                 {'name': 'chocolate syrup',
                  'quantity': 2,
                  'unit of measure': 'ounce'}],
 'prep time': 0,
 'title': 'chocolate milk',
 'uploaded_by': 'kenwalger',
 'yield': {'quantity': 1, 'unit': 'glass'}}

We have one task left, Delete. Which uses the delete_one() method. We need to pass in an argument for which document to delete which, given that our current collection only has one recipe in there we could use a variety of things to select it. Traditionally we would want to utilize the unique identifier _id to ensure that we are only deleting the document that we want.

recipes.delete_one('_id': ObjectId('588541a0146bde28a08217d4'))

Is all we need to do and our collection is empty again.

CRUD Wrap Up

In summary, we learned how to Create, Read, Update, and Delete documents in MongoDB in Python using the PyMongo driver. There is much improvement on the code to be made though by refactoring things into functions, error handling, etc. As mentioned at the top of this post, the next post will be on Bottle specifically to get up and running with that framework. After that, we will integrate MongoDB with Bottle at which time we will see some best practices for error handling and making these CRUD steps into functions.

Please let me know what you think in the comments below or reach out on Twitter where I am @kenwalger.

The code for this post can be seen in the GitHub repository here.

Happy coding!

Facebooktwittergoogle_plusredditlinkedinmail

MongoDB and Java CRUD operations

MongoDB has been out for quite a while now and with it’s 3.2 release out in Dec 2015 many of the online tutorials have become outdated as the syntax and language drivers have evolved. Let’s take a look at an introduction to the 3.2 MongoDB driver in Java. We’ll cover some real basic implementation but showcase the driver syntax and some basic MongoDB documents.

I am building this project using IntelliJ 15.0.2 and utilizing Maven for dependency management. The code is available on GitHub: here. You will want to add the following dependency to your Maven project:

<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>3.2.1</version>
</dependency>

We also will need a running MongoDB server. I will assume that you have MongoDB downloaded and installed on your local machine for the purposes of this tutorial. Our Java class will be accessing the MongoDB server on localhost, on the default port of 27017. It is also assumed that authentication for the MongoDB server is not implemented. For information on implementing access control on a MongoDB server, take a look here.

For this introductory walk-through, we are going to be doing everything inside of our main method in our Java class. In a later post we’ll see how the project can be refactored to create methods for doing our activities.

I want to show how to do a few important tasks with MongoDB. Namely,

  • Connect to a database
  • Insert individual documents into the database
  • Insert multiple documents into the database
  • Find and print documents from the database
  • Update a document.
  • Drop (delete) documents from the collection.
  • Show how to drop an entire collection.

Connect to a Database

We need to connect to specify the name of the database to use and, if the database does not exist, MongoDB creates it for us.

import com.mongodb.MongoClient;
import com.mongodb.MongoClientURI;
import com.mongodb.client.MongoDatabase;

public class MongoJava {

    public static void main(String[] args) {
        try {
            // Connect to MongoDB Server on localhost, port 27017 (default)
            final MongoClient mongoClient = new MongoClient(new MongoClientURI("mongodb://localhost:27017"));
            // Connect to Database "cartoon"
            final MongoDatabase database = mongoClient.getDatabase("cartoon");
            System.out.println("Successful database connection established. \n");

        } catch (Exception exception) {
            System.err.println(exception.getClass().getName() + ": " + exception.getMessage());
        }
    }
}

Awesome, assuming you have MongoDB up and running when we compile and run the code we will get the expected output of:

Successful database connection established.

Great! Now we need to put some data into our cartoon database. Let’s add a character to a characters collection. We do this by creating a BSON Document object, adding (appending) our data to that object, and then inserting it into the collection with the insertOne() method.

Java Crud Operations

Insert Individual Documents

We’ll create two new documents. One for information about Mickey Mouse and one for Charlie Brown.

Our import additions
import com.mongodb.ErrorCategory;
import com.mongodb.MongoWriteException;
import com.mongodb.client.MongoCollection;

import org.bson.Document;

Adding documents

            //Insert a document into the "characters" collection.
            MongoCollection collection = database.getCollection("characters");

            Document mickeyMouse = new Document();
            Document charlieBrown = new Document();

            mickeyMouse.append("_id", 1)
                    .append("characterName", "Mickey Mouse")
                    .append("creator", new Document("firstName", "Walt").append("lastName", "Disney"))
                    .append("pet", "Goofy");

            charlieBrown.append("_id", 2)
                    .append("characterName", "Charlie Brown")
                    .append("creator", new Document("firstName", "Charles").append("lastName", "Shultz"))
                    .append("pet", "Snoopy");

            try {
                collection.insertOne(mickeyMouse);
                collection.insertOne(charlieBrown);
                System.out.println("Successfully inserted documents. \n");
            } catch (MongoWriteException mwe) {
                if (mwe.getError().getCategory().equals(ErrorCategory.DUPLICATE_KEY)) {
                    System.out.println("Document with that id already exists");
                }
            }

We should expect the following output:

Successful database connection established.

Successfully inserted documents.

I know what you’re thinking. Are they really in there? We can go to the Mongo Shell by running mongo cartoon from a command prompt we will open a Mongo Shell and be using the cartoon database. We know that our data is in the characters collection so if we do a find() on our collection, and pretty print it with db.characters.find().pretty() we will get back our two documents.

{
        "_id" : 1,
        "characterName" : "Mickey Mouse",
        "creator" : {
                "firstName" : "Walt",
                "lastName" : "Disney"
        },
        "pet" : "Goofy"
}
{
        "_id" : 2,
        "characterName" : "Charlie Brown",
        "creator" : {
                "firstName" : "Charles",
                "lastName" : "Shultz"
        },
        "pet" : "Snoopy"
}

Notice how we can have documents inside documents like the creator document. Pretty cool, eh? MongoDB allows for a rich schema design in which we can embed information inside other information. Pretty powerful.

Maintenance and Multiple Inserts

This next step we’re going to do a few things. Since we are inserting our documents with specific _id values, we can’t run it again and try to again insert Mickey Mouse into our collection as we’ll generate a Duplicate Key error. So, we’ll do a bit of maintenance at the beginning of our code to delete the current characters collection each time we run our program and yet keep the data in the database at the end.

Let’s get some output about our collection size and also generate multiple documents using the insertMany() method of the driver. We’ll start with inserting documents starting at an _id of 3 (since we already have a 1 and 2) and generate enough so we have 50 documents in our collection.

Our import additions
import java.util.ArrayList;
import java.util.List;

Multiple inserts

            // Delete the collection and start fresh - add before the initial inserts
            collection.drop();

            

            // Basic data on collection
            System.out.println("Collection size: " + collection.count() + " documents. \n");

            // Create and insert multiple documents
            List documents = new ArrayList();
            for (int i = 3; i < 51; i++) {
                documents.add(new Document ("_id", i)
                        .append("characterName", "")
                        .append("creator", "")
                        .append("pet", "")
                );
            }
            collection.insertMany(documents);

            // Basic data on collection
            System.out.println("Collection size: " + collection.count() + " documents. \n");

Very nice. Now we get some output with data about our growing collection!

Successful database connection established.

Successfully inserted documents. 

Collection size: 2 documents. 

Collection size: 50 documents. 

Updating Documents

We should probably consider learning how to update a document as well. After all, we have 50 documents in the collection but only two of them have any useful data. We use the updateOne() method to update a single document along with the $set operator. We’ll print out the document with the _id of 3, update it, and print it out again to show the change. We can find our document to print using a eq Query Filter, or equals filter.

Our import additions
import com.mongodb.client.model.Filters;

Updates, find with filter

            // Update a document
            // print the third document before update.
            Document third = collection.find(Filters.eq("_id", 3)).first();
            System.out.println(third.toJson());

            collection.updateOne(new Document("_id", 3),
                    new Document("$set", new Document("characterName", "Dilbert")
                            .append("creator", new Document("firstName", "Scott").append("lastName", "Adams"))
                            .append("pet", "Dogbert"))
            );

            System.out.println("\nUpdated third document:");
            Document dilbert = collection.find(Filters.eq("_id", 3)).first();
            System.out.println(dilbert.toJson());

Nice work! Our output now is:

Successful database connection established.

Successfully inserted documents. 

Collection size: 2 documents. 

Collection size: 50 documents. 

Original third document:
{ "_id" : 3, "characterName" : "", "creator" : "", "pet" : "" }

Updated third document:
{ "_id" : 3, "characterName" : "Dilbert", "creator" : { "firstName" : "Scott", "lastName" : "Adams" }, "pet" : "Dogbert" }

Whew! Almost there. Let’s print out the entirety of our collection. We do this with a MongoCursor and iterate over all of the documents.

Our import additions
import com.mongodb.client.MongoCursor;

Find all with cursor

            // Find and print ALL documents in the collection
            System.out.println("Print the documents.");

            MongoCursor cursor = collection.find().iterator();
            try {
                while (cursor.hasNext()) {
                    System.out.println(cursor.next().toJson());
                }

            } finally {
                cursor.close();
            }

With all of the new output, and most of it empty, perhaps we should clean up our collection a bit. We currently have 50 documents in there, but only three of them have any pertinent data. Let’s get rid of all documents that don’t have data using the deleteMany() function. We’re going to need another filter here to delete documents whose _id is greater than or equal to 4. Fortunately there is a built in filter, gte that will do just that!

Our final code for this tutorial then is available here as a gist.

            //Delete data
            System.out.println("\nDelete documents with an id greater than or equal to 4.");
            collection.deleteMany(Filters.gte("_id", 4));

            // Find and print ALL documents in the collection
            System.out.println("\nPrint all documents.");

            MongoCursor cursor2 = collection.find().iterator();
            try {
                while (cursor2.hasNext()) {
                    System.out.println(cursor2.next().toJson());
                }

            } finally {
                cursor2.close();
            }

As we would expect, our collection is now left with only three documents, one each for Mickey Mouse, Charlie Brown, and Dilbert. Our final code for this tutorial then is available here as a gist.

Summary

We have learned quite a bit. We have seen how to connect to an active MongoDB server, create new documents, update documents, find documents in our collection, and delete documents. In the database world that is called CRUD, which is an acronym for Create, Read, Update, Delete. Pretty great.

Happy coding!

Facebooktwittergoogle_plusredditlinkedinmail