{"id":347,"date":"2017-06-06T05:49:19","date_gmt":"2017-06-06T12:49:19","guid":{"rendered":"http:\/\/www.kenwalger.com\/blog\/?p=347"},"modified":"2017-08-03T05:38:01","modified_gmt":"2017-08-03T12:38:01","slug":"mongodb-horizontal-scaling-sharding","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/nosql\/mongodb\/mongodb-horizontal-scaling-sharding\/","title":{"rendered":"MongoDB Horizontal Scaling through Sharding"},"content":{"rendered":"

There comes a time in many MongoDB<\/a> database, or any database for that matter, life cycles in which our data outgrows our servers. Either physically outgrows storage capabilities or the data grows so large that performance is degraded. Even scaling our physical servers up with a more powerful CPU, more RAM, or hard drives (vertical scaling) may not be enough. This is where horizontal scaling through sharding comes into practice.<\/p>\n

Sharding<\/h3>\n

What is sharding exactly? It is the practice of distributing data across multiple machines. In MongoDB it supports instances with large data sets and needed high throughput operations. The data is distributed across all shards allowing the workload to be evenly shared. This has the potential for much better efficiency than a single server.<\/p>\n

\"Sharded<\/a><\/p>\n

Each cluster should, for redundancy, also be replica sets<\/a> with primary<\/a> and secondary<\/a> servers.<\/p>\n

\"Sharded<\/a><\/p>\n

MongoDB\u00a0data is sharded at a collection<\/a> level, therefore it isn’t necessary to distribute the entire database across a sharded environment.<\/p>\n

Cluster Configuration<\/h3>\n

There are three components to a sharded cluster<\/a> that we need. We need the shards themselves, a query router in the way of a mongos<\/a> server, and configuration or config servers<\/a>.<\/p>\n

The shards are what store the subset of the data. The config servers store all of the metadata about the cluster and which shard houses what data. Data is stored in chunks<\/a> on each shard and the config server keeps track of all that information.<\/p>\n

The mongos, in its role as a query router, acts as the interface between the application and the data. It routes queries and write operations to the appropriate shards. An application, therefore, will only access the data\u00a0through<\/em> a mongos, never by touching the data itself. Queries are routed via the mongos to all shards\u00a0unless<\/strong> it can be determined the data resides on a particular shard.<\/p>\n

\"Broadcast<\/a><\/p>\n

There has to be a better way than broadcasting that request to\u00a0all<\/strong> shards though, right? As you might imagine, this “scatter\/gather” approach to querying can result in some long running operations. Well, I kind of eluded to it before by qualifying it with the “unless”, so there is a better way! Enter the shard key<\/a>.<\/p>\n

Shard Keys<\/h3>\n

A shard key determines how documents<\/a> in a collection are distributed across the shards. It is an indexed field, or indexed compound fields, that exists in\u00a0every<\/strong> document in the collection. Recall that MongoDB allows for a flexible schema<\/a>\u00a0within the document model<\/a>. This is one consideration when choosing a shard key; every document must have the indexed field.<\/p>\n

If provided with a shard key during a query, the mongos knows how to route the request.<\/p>\n

\"Targeted<\/a><\/p>\n

This can greatly enhance performance. Choosing a good shard key, however, is very important. I’ve covered what goes into the selection of a good shard key in a different post<\/a>.<\/p>\n

Trade offs to Sharding<\/h3>\n

As the saying goes, there’s no such thing as a free lunch. Sharding is the same way. Sharding your data sets increases infrastructure complexity as well as maintenance. One solution to help mitigate both of these is to utilize a DBaaS such as Atlas<\/a>\u00a0to host your MongoDB data.<\/p>\n

If queries are run without including a shard key, the “scatter\/gather” approach is used. This can result in slow queries, therefore it is definitely something to remember when writing your applications.<\/p>\n

Once a collection is sharded, it cannot be unsharded. Similarly, once a shard key is selected, it cannot be changed. So these steps need to be undertaken with careful planning.<\/p>\n

If you are handling things yourself on your own hardware\u00a0I have briefly discussed some of the tools which can be used to check performance of a sharded collection in previous posts. Specifically in MongoDB CLI Tools<\/a> and briefly in MongoDB explain() explained<\/a>.<\/p>\n

Wrap Up<\/h3>\n

When your data has outgrown a single server, sharding is a great approach to keep your database performing well. There are some things to watch out for though. Make sure you choose a good shard key and stay up to date with database maintenance.<\/p>\n

There are a lot of MongoDB specific terms in this post. I created a MongoDB Dictionary<\/a> skill for the Amazon Echo<\/a>\u00a0line of products. Check it out and you can say “Alexa, ask MongoDB what is a Shard?” and get a helpful response.<\/p>\n

\n

Follow me on Twitter @kenwalger<\/a> to get the latest updates on my postings.<\/em><\/p>\n\"Facebook\"<\/a>\"twitter\"<\/a>\"reddit\"<\/a>\"linkedin\"<\/a>\"mail\"<\/a>","protected":false},"excerpt":{"rendered":"

There comes a time in many MongoDB database, or any database for that matter, life cycles in which our data outgrows our servers. Either physically outgrows storage capabilities or the data grows so large that performance is degraded. Even scaling our physical servers up with a more powerful CPU, more RAM, or hard drives (vertical … Continue reading “MongoDB Horizontal Scaling through Sharding”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":348,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pmpro_default_level":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4,3],"tags":[61],"yst_prominent_words":[117,487,99,739,104,486,474,483,737,87,489,488,738,93,736,469,741,485,740,477],"class_list":["post-347","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mongodb","category-nosql","tag-sharding","pmpro-has-access"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/uploads\/2017\/06\/sharding-e1496717243397.png?fit=125%2C125&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8lx70-5B","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/347","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/comments?post=347"}],"version-history":[{"count":5,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/347\/revisions"}],"predecessor-version":[{"id":490,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/347\/revisions\/490"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media\/348"}],"wp:attachment":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media?parent=347"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/categories?post=347"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/tags?post=347"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/yst_prominent_words?post=347"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}