Author: admin

A Review of PyCon 2017

PyCon 2017 was held about 45 miles south of Portland, Oregon where I am fortunate enough to live. I am typically not a great conference attendee so I was a bit nervous and apprehensive about going. After walking into the Portland Convention Center and being surrounded by 3,500 fellow Python enthusiasts, however, I was super excited.

PyCon 2017 Talks

There were several amazing talks on a wide array of subjects. There were first time PyCon presenters like Jonas Neubert with a fascinating talk on using Python for factory automation. Along with some “heavy hitters” in the Python community such as Philip James and Daniele Procida. Daniele’s talk on documentation was very interesting and definitely a different way of thinking of of the documentation process.

There were some really good keynote speeches as well. I really enjoyed learning from Lisa Guo on the migration path Instagram took to go from Python 2 to Python 3. I also found Dr. Katy Huff‘s talk on some practical applications for Python in the science field to be quite interesting.

Kenneth Love gave a tutorial on the Django Admin which I would encourage people to work through as well. As usual it is a great presentation and Mr. Love provides excellent information.

There were many other great talks, from many great speakers. The conference talks are available on YouTube here. I’d encourage you to watch, listen, and learn from as many as possible.

Vendors

There were some great vendors and businesses in attendance. Big companies like Google, Microsoft, Intel, and Facebook/Instagram were all there. They all offered some excellent short talks in their booths on how they were using Python for their applications or products. They also had a variety of swag they were giving away in exchange for a badge scan.

Google had an hourly trivia session and provided prizes for the correct answer. I picked up a copy of Python in a Nutshell: A Desktop Quick Reference for knowing that PyCon 2018 would be held in Cleveland, Ohio. Intel had several drawings for an Amazon Echo, along with several other vendors actually. Booz Allen Hamilton had several drawings for a Raspberry PI 3 Model B which I sadly wasn’t able to win.

Company branded socks were one of the big swag things at PyCon, beyond the t-shirts, stickers, and fidget spinners. Microsoft, O’Reilly Publishing, Citus Data, and Heroku all had custom socks. Scan your badge… get socks.

PyCon 2017 Vendor socks

Community

In addition to the great learning opportunities the talks provided, and the interaction with sponsors, another key feature of any conference is the people. The opportunity to get to meet and talk with people like Andrew Pinkham who authored Django Unleashed, or Russell Keith-Magee of BeeWare was spectacular. Having the chance to meet and talk with other developers about industry trends was great as well.

I have heard that the Python community is second to none in terms of inclusiveness and I was able to witness first hand that is indeed the case. Due to the exceptional overall experience I had at PyCon 2017, I am sincerely hoping that I can make to the trip to Cleveland in 2018.

Facebooktwitterredditlinkedinmail

MongoDB explain() explained

There are many different considerations to be made when running queries in MongoDB. A helpful thing to use in the mongo shell when running a find() operation is to use the explain() method. In this blog post, I’ll take a look at some of the options for explain() and what the results mean.

explain()

As discussed in a previous post on indexing in MongoDB, we can use the explain() method to learn about the selected query plan. This allows for an examination of the performance of a given query. It can be used in the following manner:

db.collection.find().explain()

The information generated can be used to see what index is being used for a query, if the query is a covered query,  and which servers in a sharded collection the query is run against, to name a few.

Three different verbosity modes can be utilized to determine the amount of information provided.

Verbosity modes
  • queryPlanner – the given query provided in the find() method is put through the query optimizer to find the most efficient query. This “winning plan” is then passed to the queryPlanner and the information is returned for the evaluated query. The query is not run in this mode. As a result things like query time, e.g. executionTimeMillisEstimate are true estimates since the query has not been executed.
  • executionStats – when running in this mode, the query optimizer is run and the query is fully executed. The information returned details the results of the are what actually happened during that specific query.
  • allPlansExecution – as the name might suggest, this mode returns information about all possible query plans. While the winning plan is executed and statistics returned for it, other candidate plan information is returned as well. This is the default mode of explain().

The variety of information these different modes provides can be extremely useful. Let’s take a look at some returned results of explain() and walk through what they show.

Results

For this example, I will use a test example database of a blog. The database contains two collections, users and articles, and is running on a single, unsharded, machine. Each collection has, roughly, 550,500 documents and is not indexed beyond the index for _id.

Let’s start with looking at what gets returned from a query for a single username. And take a look at some of the bits and pieces of information provided.

db.users.find( { "username": "User_9"} ).explain()

 

explain output

The parsedQuery section is the query we are exploring. The query stage provides a description of the type of operation that occurred for the winning plan.

Operation Types

  • COLLSCAN – indicates a collection scan occurred for the query, meaning that the query looked at each document to get the results
  • IXSCAN – indicates an index was used for the query
  • FETCH – for retrieving documents
  • SHARD_MERGE – the result of merging data from shards

The stage is a tree structure and can have multiple, child, stages. The direction of the query shows whether the query was performed in a forward or reverse order. The serverInfo section displays information on the server the query was run against and includes, in the version key, the version of the MongoDB database. If the collection was in a sharded environment, each accessed shard would be listed in the serverInfo.

When the command is run using the “executionStats” verbosity mode:

db.users.find({ "username": "User_9"} ).explain("executionStats")

additional information is provided as a result of the query being run on the data.

explain with executionStats

Here we see, among other things, the time the query took to run, along with how many documents were returned, nReturned, and how many documents were examined by the database, totalDocsExamined. As mentioned in my post on indexing, ideally these two numbers should be very close to the same value.

Wrap Up

There is a lot of information available when using the explain() method. It provides some great information about how queries are actually being run and gives an indication as to where a collection can benefit from an index. It should be your first stop when examining slow queries before moving onto other MongoDB tools.

There are a lot of MongoDB specific terms in this post. I created a MongoDB Dictionary skill for the Amazon Echo line of products. Check it out and you can say “Alexa, ask MongoDB what is an index?” and get a helpful response.

Follow me on Twitter @kenwalger to get the latest updates on my postings.

Facebooktwitterredditlinkedinmail