{"id":601,"date":"2017-10-13T08:09:41","date_gmt":"2017-10-13T15:09:41","guid":{"rendered":"https:\/\/www.kenwalger.com\/blog\/?p=601"},"modified":"2017-10-13T08:09:41","modified_gmt":"2017-10-13T15:09:41","slug":"big-data-nosql-great-coupling","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/nosql\/big-data-nosql-great-coupling\/","title":{"rendered":"Big Data and NoSQL: A Great Coupling"},"content":{"rendered":"<p>Big data has been a buzzword and topic of conversation for many years. Similarly, the idea of NoSQL as a data store versus &#8220;traditional&#8221; relational databases is a frequently debated topic. With all of the conversation about these topics, it is important to ask a few questions. What is this thing called big data? Are NoSQL databases like <a href=\"http:\/\/orientdb.com\">OrientDB<\/a> a good fit for big data? Let&#8217;s explore these questions and learn why NoSQL can indeed be a\u00a0<strong>great<\/strong> fit for big data.<\/p>\n<h3>Big Data Defined<\/h3>\n<p>A popular definition of big data states:<\/p>\n<blockquote><p>Any quantity of data which is too large or complex that traditional computing power is inadequate to deal with.<\/p><\/blockquote>\n<p>While this makes sense, it is important to note that the concept of big data is\u00a0<em>somewhat<\/em> relative. It likely has a much different meaning and concept for Farmer Bob, the local apple farmer than it does for <a href=\"http:\/\/www.treetop.com\/\">Tree Top<\/a> juice company. In the technology realm, we often think of companies like Facebook or Netflix in relationship to big data processing. But every industry is capturing more and more data.<\/p>\n<p>Think about our apple farmers. Some of the data they might want to be collecting would be soil moisture levels, air &amp; soil temperatures, rainfall, the type of apple being grown, the geo-location of the trees, soil pH, fertilizer applications, pollination\u00a0events, just to name a few.<\/p>\n<p>In the data industry, there are four common dimensions of big data. The so-called\u00a0<a href=\"http:\/\/www.ibmbigdatahub.com\/infographic\/four-vs-big-data\">4 V&#8217;s<\/a>, volume, variety, velocity, and veracity.<\/p>\n<h5>4 Dimensions of Big Data<\/h5>\n<ul>\n<li>Volume &#8211; the size and scale of the data, a terabyte or more of data is often a breakpoint. But even that is relatively small. A Boeing 737 generates approximately 240 terabytes of data flying across the United States. How &#8217;bout them apples?<\/li>\n<li>Variety &#8211; the diversity of data in different forms. In our farming example, there could be different sensors around the orchard sending a variety of data into the database.<\/li>\n<li>Velocity &#8211; the speed at which data is collected and needs to be processed. Perhaps not entirely applicable for Farmer Bob, but for many big data applications velocity is very important. Stock trading algorithms change very quickly and must be timely, for example.<\/li>\n<li>Veracity &#8211; the quality of data itself, is it accurate? Is it stored accurately and can it be retrieved accurately?<\/li>\n<\/ul>\n<p>All of these pieces together help to influence the decision as to if you are dealing with big data.<\/p>\n<p>Assuming, then, that our <a href=\"https:\/\/www.amazon.com\/gp\/product\/0375847448\/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=0375847448&amp;linkCode=as2&amp;tag=kenwalgersite-20&amp;linkId=628bb05aa0ea2038daa55043d53bec88\" target=\"_blank\" rel=\"noopener\">apple orchard<\/a><img loading=\"lazy\" decoding=\"async\" style=\"border: none !important; margin: 0px !important;\" src=\"\/\/ir-na.amazon-adsystem.com\/e\/ir?t=kenwalgersite-20&amp;l=am2&amp;o=1&amp;a=0375847448\" alt=\"\" width=\"1\" height=\"1\" border=\"0\" \/> is large enough to qualify for big data membership, why would we choose a NoSQL database? How does a NoSQL database, like OrientDB, help to address the concerns of the 4 V&#8217;s?<\/p>\n<h3>Stop the Presses: NoSQL for the Win<\/h3>\n<p>Okay, a bit of a pun there with <a href=\"https:\/\/www.amazon.com\/gp\/product\/B01154QQIY\/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=B01154QQIY&amp;linkCode=as2&amp;tag=kenwalgersite-20&amp;linkId=ec82160d3318edf8f91af407a80f5618\" target=\"_blank\" rel=\"noopener\">presses<\/a><img loading=\"lazy\" decoding=\"async\" style=\"border: none !important; margin: 0px !important;\" src=\"\/\/ir-na.amazon-adsystem.com\/e\/ir?t=kenwalgersite-20&amp;l=am2&amp;o=1&amp;a=B01154QQIY\" alt=\"\" width=\"1\" height=\"1\" border=\"0\" \/> while talking about apple orchards, please forgive me. But seriously, NoSQL does offer some great wins for storing big data. In fact, in many of the areas of the 4 V&#8217;s NoSQL is a better choice than its relational, SQL, counterpart.<\/p>\n<p>The variety of data becomes problematic in a relational data model as it requires multiple tables and many Joins to process the data. A multi-model database like OrientDB is capable of storing records in a schemaless fashion that is easy to generate programmatically. This allows for data with a variety of &#8220;looks&#8221; to be stored quickly into a single <a href=\"http:\/\/orientdb.com\/docs\/last\/Concepts.html#record\">record<\/a>. Being a multi-model database, a variety of types of records can be stored in the same <a href=\"http:\/\/orientdb.com\/docs\/last\/Concepts.html#database\">database<\/a>.<\/p>\n<p>Further, without having to do Joins, traversing the data for read operations is very fast.\u00a0The ability to read and write records rapidly is vital for keeping up with the velocity of data. Performance of the database is a huge portion of being able to reliably process your data and get results.<\/p>\n<p>One of the challenges that SQL databases face is that of scalability. They are mostly designed around a master-slave architecture which often becomes a bottleneck. The solution to scale is over to upgrade hardware to bigger and faster. In the NoSQL world, we can scale with a multi-master and sharded architecture. This allows for horizontal scaling and adds to veracity as well as the new servers can be spread out across geographic locations.<\/p>\n<p>Making sure the data you\u00a0<em>think<\/em> you are storing is actually being stored is probably the most important of all. Having the ability and confidence of knowing that your database transactions are secure and accurate is huge. OrientDB <a href=\"http:\/\/orientdb.com\/docs\/last\/Transactions.html\">transactions<\/a> are ACID compliant providing an extra level of guarantee that your data is your data. This also helps to make sure that the technical department doesn&#8217;t upset the apple cart with corporate executives.<\/p>\n<h3>Wrap Up<\/h3>\n<p>We have taken a look at what big data is and how to determine if your data\u00a0<em>is<\/em><strong>\u00a0<\/strong>big data. We then examined why NoSQL is a great fit for storing our data. OrientDB is definitely worth a look at storing your NoSQL data for your big data needs.<\/p>\n<hr \/>\n<p><em>Follow me on Twitter <a href=\"https:\/\/www.twitter.com\/kenwalger\">@kenwalger<\/a> to get the latest updates on my postings or read my other posts on <a href=\"https:\/\/www.kenwalger.com\/blog\">my blog<\/a>.<\/em><\/p>\n<a class=\"synved-social-button synved-social-button-share synved-social-size-48 synved-social-resolution-single synved-social-provider-facebook nolightbox\" data-provider=\"facebook\" target=\"_blank\" rel=\"nofollow\" title=\"Share on Facebook\" href=\"https:\/\/www.facebook.com\/sharer.php?u=https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601&#038;t=Big%20Data%20and%20NoSQL%3A%20A%20Great%20Coupling&#038;s=100&#038;p&#091;url&#093;=https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601&#038;p&#091;images&#093;&#091;0&#093;=https%3A%2F%2Fi0.wp.com%2Fwww.kenwalger.com%2Fblog%2Fwp-content%2Fuploads%2F2017%2F10%2FOrientDB_blog_feature.png%3Ffit%3D125%252C125%26ssl%3D1&#038;p&#091;title&#093;=Big%20Data%20and%20NoSQL%3A%20A%20Great%20Coupling\" style=\"font-size: 0px;width:48px;height:48px;margin:0;margin-bottom:5px;margin-right:5px\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" alt=\"Facebook\" title=\"Share on Facebook\" class=\"synved-share-image synved-social-image synved-social-image-share\" width=\"48\" height=\"48\" style=\"display: inline;width:48px;height:48px;margin: 0;padding: 0;border: none;box-shadow: none\" src=\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/facebook.png?resize=48%2C48&#038;ssl=1\" \/><\/a><a class=\"synved-social-button synved-social-button-share synved-social-size-48 synved-social-resolution-single synved-social-provider-twitter nolightbox\" data-provider=\"twitter\" target=\"_blank\" rel=\"nofollow\" title=\"Share on Twitter\" href=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601&#038;text=Hey%20check%20this%20out\" style=\"font-size: 0px;width:48px;height:48px;margin:0;margin-bottom:5px;margin-right:5px\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" alt=\"twitter\" title=\"Share on Twitter\" class=\"synved-share-image synved-social-image synved-social-image-share\" width=\"48\" height=\"48\" style=\"display: inline;width:48px;height:48px;margin: 0;padding: 0;border: none;box-shadow: none\" src=\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/twitter.png?resize=48%2C48&#038;ssl=1\" \/><\/a><a class=\"synved-social-button synved-social-button-share synved-social-size-48 synved-social-resolution-single synved-social-provider-reddit nolightbox\" data-provider=\"reddit\" target=\"_blank\" rel=\"nofollow\" title=\"Share on Reddit\" href=\"https:\/\/www.reddit.com\/submit?url=https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601&#038;title=Big%20Data%20and%20NoSQL%3A%20A%20Great%20Coupling\" style=\"font-size: 0px;width:48px;height:48px;margin:0;margin-bottom:5px;margin-right:5px\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" alt=\"reddit\" title=\"Share on Reddit\" class=\"synved-share-image synved-social-image synved-social-image-share\" width=\"48\" height=\"48\" style=\"display: inline;width:48px;height:48px;margin: 0;padding: 0;border: none;box-shadow: none\" src=\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/reddit.png?resize=48%2C48&#038;ssl=1\" \/><\/a><a class=\"synved-social-button synved-social-button-share synved-social-size-48 synved-social-resolution-single synved-social-provider-linkedin nolightbox\" data-provider=\"linkedin\" target=\"_blank\" rel=\"nofollow\" title=\"Share on Linkedin\" href=\"https:\/\/www.linkedin.com\/shareArticle?mini=true&#038;url=https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601&#038;title=Big%20Data%20and%20NoSQL%3A%20A%20Great%20Coupling\" style=\"font-size: 0px;width:48px;height:48px;margin:0;margin-bottom:5px;margin-right:5px\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" alt=\"linkedin\" title=\"Share on Linkedin\" class=\"synved-share-image synved-social-image synved-social-image-share\" width=\"48\" height=\"48\" style=\"display: inline;width:48px;height:48px;margin: 0;padding: 0;border: none;box-shadow: none\" src=\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/linkedin.png?resize=48%2C48&#038;ssl=1\" \/><\/a><a class=\"synved-social-button synved-social-button-share synved-social-size-48 synved-social-resolution-single synved-social-provider-mail nolightbox\" data-provider=\"mail\" rel=\"nofollow\" title=\"Share by email\" href=\"mailto:?subject=Big%20Data%20and%20NoSQL%3A%20A%20Great%20Coupling&#038;body=Hey%20check%20this%20out:%20https%3A%2F%2Fwww.kenwalger.com%2Fblog%2Fwp-json%2Fwp%2Fv2%2Fposts%2F601\" style=\"font-size: 0px;width:48px;height:48px;margin:0;margin-bottom:5px\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" alt=\"mail\" title=\"Share by email\" class=\"synved-share-image synved-social-image synved-social-image-share\" width=\"48\" height=\"48\" style=\"display: inline;width:48px;height:48px;margin: 0;padding: 0;border: none;box-shadow: none\" src=\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/mail.png?resize=48%2C48&#038;ssl=1\" \/><\/a>","protected":false},"excerpt":{"rendered":"<p>Big data has been a buzzword and topic of conversation for many years. Similarly, the idea of NoSQL as a data store versus &#8220;traditional&#8221; relational databases is a frequently debated topic. With all of the conversation about these topics, it is important to ask a few questions. What is this thing called big data? Are &hellip; <a href=\"https:\/\/www.kenwalger.com\/blog\/nosql\/big-data-nosql-great-coupling\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Big Data and NoSQL: A Great Coupling&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":648,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pmpro_default_level":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[965,966],"yst_prominent_words":[937,933,922,104,958,956,953,926,944,962,923,934,935,963,840,790,945,930,957,931],"class_list":["post-601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-nosql","tag-big-data","tag-orientdb","pmpro-has-access"],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/uploads\/2017\/10\/OrientDB_blog_feature.png?fit=125%2C125&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8lx70-9H","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/comments?post=601"}],"version-history":[{"count":6,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/601\/revisions"}],"predecessor-version":[{"id":647,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/601\/revisions\/647"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media\/648"}],"wp:attachment":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media?parent=601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/categories?post=601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/tags?post=601"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/yst_prominent_words?post=601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}