For a small dataset, everything feels manageable.<\/p>\n
A few rocks here and there.<\/p>\n
A handful of records.<\/p>\n
It\u2019s easy to reason about the system.<\/p>\n
When the Dataset Grows<\/h2>\n
The moment the dataset starts to grow, the assumptions change.<\/p>\n
Instead of a few rocks, imagine:<\/p>\n
- \n
- hundreds<\/li>\n
- thousands<\/li>\n
- eventually, many thousands<\/li>\n<\/ul>\n
At that point, a few new questions appear:<\/p>\n
- \n
- How do we process incoming data efficiently?<\/li>\n
- Where do we store large assets?<\/li>\n
- How do we keep queries fast?<\/li>\n
- What happens when processing takes longer than capture?<\/li>\n<\/ul>\n
These are the same questions that show up in any system dealing with real-world data.<\/p>\n
The Pipeline Becomes the System<\/h2>\n
At small scale, the pipeline is implicit.<\/p>\n
You take a photo.<\/p>\n
You upload it.<\/p>\n
You update a record.<\/p>\n
At larger scale, that approach breaks down.<\/p>\n
The pipeline becomes explicit.<\/p>\n
![\"Diagram](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/uploads\/2026\/04\/physical-object-data-pipeline-scalable-architecture.png?resize=680%2C763&ssl=1\")
At scale, simple data flows evolve into multi-stage pipelines with decoupled processing and storage.<\/figcaption><\/figure>\n Each stage now has a role:<\/p>\n
- \n
- capture generates raw input<\/li>\n
- ingestion buffers incoming data<\/li>\n
- processing transforms it<\/li>\n
- storage persists it<\/li>\n
- indexing makes it usable<\/li>\n<\/ul>\n
What used to be a simple flow becomes a system of components.<\/p>\n
Decoupling the System<\/h2>\n
One of the first things that happens at scale is decoupling.<\/p>\n
Instead of doing everything at once, we separate concerns:<\/p>\n
- \n
- capture does not block processing<\/li>\n
- processing does not block storage<\/li>\n
- storage does not block indexing<\/li>\n<\/ul>\n
This introduces queues and asynchronous work.<\/p>\n
Instead of:<\/p>\n
take photo \u2192 process \u2192 store \u2192 done\n<\/code><\/pre>\nwe now have:<\/p>\n
take photo \u2192 enqueue \u2192 process later \u2192 update system\n<\/code><\/pre>\nThis improves resilience.<\/p>\n
It also introduces complexity.<\/p>\n
Storage Starts to Matter<\/h2>\n
At small scale, storage decisions are easy.<\/p>\n
At larger scale, they matter.<\/p>\n
We now have different types of data:<\/p>\n
- \n
- metadata (small, structured)<\/li>\n
- images (large, unstructured)<\/li>\n
- 3D models (larger, computationally expensive to generate)<\/li>\n<\/ul>\n
These tend to be stored differently:<\/p>\n
- \n
- database for structured data<\/li>\n
- object storage for assets<\/li>\n
- references connecting the two<\/li>\n<\/ul>\n
This separation becomes critical for performance and cost.<\/p>\n
Processing Becomes a Bottleneck<\/h2>\n
Not all steps in the pipeline are equal.<\/p>\n
Some are fast:<\/p>\n
- \n
- inserting metadata<\/li>\n
- updating records<\/li>\n<\/ul>\n
Others are slow:<\/p>\n
- \n
- generating 3D models<\/li>\n
- running image processing<\/li>\n
- extracting features<\/li>\n<\/ul>\n
As the dataset grows, these slower steps become bottlenecks.<\/p>\n
Which leads to another pattern:<\/p>\n
Parallelization<\/strong>.<\/p>\n
Instead of one process handling everything, we distribute the work.<\/p>\n
Multiple workers.<\/p>\n
Multiple jobs.<\/p>\n
Multiple stages running simultaneously.<\/p>\n
Indexing at Scale<\/h2>\n
Search also changes at scale.<\/p>\n
At small scale:<\/p>\n
- \n
- simple queries are fast<\/li>\n
- no special indexing required<\/li>\n<\/ul>\n
At larger scale:<\/p>\n
- \n
- indexes must be built and maintained<\/li>\n
- similarity search requires preprocessing<\/li>\n
- updates must propagate through the system<\/li>\n<\/ul>\n
Search becomes an active part of the pipeline, not just a query on top of it.<\/p>\n
Failure Becomes Normal<\/h2>\n
At small scale, failures are rare and easy to fix.<\/p>\n
At larger scale, failures are expected.<\/p>\n
Examples:<\/p>\n
- \n
- missing images<\/li>\n
- failed processing jobs<\/li>\n
- incomplete models<\/li>\n
- inconsistent metadata<\/li>\n<\/ul>\n
The system must tolerate these failures.<\/p>\n
Not eliminate them.<\/p>\n
This leads to:<\/p>\n
- \n
- retries<\/li>\n
- partial results<\/li>\n
- eventual consistency<\/li>\n<\/ul>\n
In other words, the system becomes more realistic.<\/p>\n
A Familiar Architecture<\/h2>\n
At this point, the Backyard Quarry<\/strong> starts to resemble a typical data platform.<\/p>\n
![\"Layered](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/uploads\/2026\/04\/pyhsical-to-digital-system-architecture-layers.png?resize=286%2C1024&ssl=1\")
A common architectural pattern for systems that transform physical inputs into digital data.
<\/code><\/figcaption><\/figure>\nDifferent domains implement this differently.<\/p>\n
But the structure is remarkably consistent.<\/p>\n
The Tradeoff<\/h2>\n
Scaling introduces tradeoffs.<\/p>\n
We gain:<\/p>\n
- \n
- throughput<\/li>\n
- flexibility<\/li>\n
- resilience<\/li>\n<\/ul>\n
We lose:<\/p>\n
- \n
- simplicity<\/li>\n
- immediacy<\/li>\n
- ease of reasoning<\/li>\n<\/ul>\n
What was once a straightforward system becomes a collection of interacting parts.<\/p>\n
The Real Shift<\/h2>\n
The most important change isn\u2019t technical.<\/p>\n
It\u2019s conceptual.<\/p>\n
At small scale, you think about individual objects.<\/p>\n
At larger scale, you think about systems.<\/p>\n
You stop asking:<\/p>\n
How do I store this rock?<\/em><\/p>\n
And start asking:<\/p>\n
How does the system handle many rocks over time?<\/em><\/p>\n
That shift is what turns a project into a platform.<\/p>\n
What Comes Next<\/h2>\n
At this point, the Backyard Quarry<\/strong> is no longer just a small experiment.<\/p>\n
It\u2019s a miniature version of a data platform.<\/p>\n
And the patterns we\u2019ve seen \u2014 schema design, pipelines, indexing, scaling \u2014 show up in many places.<\/p>\n
In the next post, we\u2019ll zoom out even further.<\/p>\n
Because once you start recognizing these patterns, you begin to see them everywhere.<\/p>\n
Not just in rock piles.<\/p>\n
But in systems across industries.<\/p>\n
And somewhere along the way, the Quarry stopped being about rocks.<\/p>\n
It became about how systems grow.<\/p>\n
The Rock Quarry Series<\/h2>\n
- \n
- The Backyard Quarry: Turning Rocks Into Data<\/a><\/li>\n
- The Backyard Quarry, Part 2: Designing a Schema for Physical Objects<\/a><\/li>\n
- The Backyard Quarry, Part 3: Capturing the Physical World<\/a><\/li>\n
- The Backyard Quarry, Part 4: Searching a Pile of Rocks<\/a><\/li>\n
- The Backyard Quarry, Part 5: Digital Twins for Physical Objects<\/a><\/li>\n<\/ul>\n
![\"Facebook\"](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/facebook.png?resize=48%2C48&ssl=1\" "\\"Share")
<\/a>![\"twitter\"](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/twitter.png?resize=48%2C48&ssl=1\" "\\"Share")
<\/a>![\"reddit\"](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/reddit.png?resize=48%2C48&ssl=1\" "\\"Share")
<\/a>![\"linkedin\"](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/linkedin.png?resize=48%2C48&ssl=1\" "\\"Share")
<\/a>![\"mail\"](\"https:\/\/i0.wp.com\/www.kenwalger.com\/blog\/wp-content\/plugins\/social-media-feather\/synved-social\/image\/social\/regular\/96x96\/mail.png?resize=48%2C48&ssl=1\" "\\"Share")
<\/a>","protected":false},"excerpt":{"rendered":"So far, the Backyard Quarry system has worked well. We have: a schema a capture process stored assets searchable data digital twins For a small dataset, everything feels manageable. A few rocks here and there. A handful of records. It\u2019s easy to reason about the system. When the Dataset Grows The moment the dataset starts … Continue reading “The Backyard Quarry, Part 6: Scaling the Quarry”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pmpro_default_level":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1739,1738],"tags":[1736,1748,1756,1755,1713],"yst_prominent_words":[99,964,575,797],"class_list":["post-1358","post","type-post","status-publish","format-standard","hentry","category-data-engineering","category-software-engineering","tag-data-engineering","tag-data-pipelines","tag-distributed-systems","tag-scalability","tag-system-design","pmpro-has-access"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8lx70-lU","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1358","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/comments?post=1358"}],"version-history":[{"count":6,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1358\/revisions"}],"predecessor-version":[{"id":1366,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1358\/revisions\/1366"}],"wp:attachment":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media?parent=1358"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/categories?post=1358"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/tags?post=1358"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/yst_prominent_words?post=1358"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- The Backyard Quarry, Part 2: Designing a Schema for Physical Objects<\/a><\/li>\n
- The Backyard Quarry: Turning Rocks Into Data<\/a><\/li>\n