What I’ve Been Building: Systems, AI, and Real-World Data

Over the past several weeks, I’ve been spending a lot of time thinking about systems.

Some of that thinking has taken the form of writing.

If you’ve come across any of my recent posts, they might seem like they cover very different topics:

  • cataloging rocks in a backyard
  • building AI systems using MCP
  • working with documents, images, and real-world data

At first glance, they don’t appear to have much in common.

But they’re all exploring the same underlying idea.

The Common Thread

Across all of these posts, the focus has been on a specific kind of problem:

How do we turn messy, real-world inputs into structured, usable systems?

That problem shows up in many different forms.

Sometimes the input is physical:

  • objects
  • artifacts
  • environments

Sometimes it’s digital:

  • documents
  • images
  • logs

Sometimes it’s dynamic:

  • motion
  • behavior
  • sensor data

But the challenge is the same.

The input is unstructured.

The system needs structure.

The Backyard Quarry

One way I explored this idea was through a small project I called the Backyard Quarry.

It started with a simple observation:

There are a lot of rocks in the yard.

From there, the problem evolved into something more interesting:

  • how to represent physical objects as data
  • how to capture images and measurements
  • how to build pipelines around that data
  • how to search and organize it
  • how to think about digital twins

What began as a small experiment became a way to explore system design in a constrained, tangible setting.

MCP and AI Systems

In parallel, I’ve been writing about building AI systems using MCP.

On the surface, this looks very different.

Instead of rocks, the inputs are:

  • documents
  • APIs
  • models
  • agent workflows

But the structure is familiar.

  • inputs are ingested
  • processed
  • transformed
  • routed
  • used by applications

The system still needs to handle:

  • variability
  • scale
  • imperfect data
  • orchestration

Different inputs.

Same patterns.

From Objects to Systems

One of the more useful realizations in working through these ideas is this:

The problem is rarely about the individual object.
It’s about the system that handles many objects over time.

Whether the object is:

  • a rock
  • a document
  • a sensor reading

The questions become:

  • how is it represented?
  • how does it enter the system?
  • how is it transformed?
  • how is it stored?
  • how is it retrieved?

These are system-level questions.

A Shared Architecture

Across these different domains, a common architecture begins to emerge.

Diagram showing how raw inputs are captured, processed, structured, indexed, and used by applications in a data system.
A common pattern for transforming real-world inputs into usable systems.

The labels change depending on the domain.

But the structure remains consistent.

Why This Matters

Understanding this pattern makes it easier to approach new problems.

Instead of starting from scratch each time, you can ask:

  • Where does the data come from?
  • How does it enter the system?
  • What transformations are required?
  • How will it be used?

This reduces complexity.

It also makes systems more predictable.

What I’m Interested In

Going forward, I’m particularly interested in systems that sit at the boundary between:

  • the physical world and digital systems
  • unstructured inputs and structured data
  • human workflows and automated processes

That includes areas like:

  • digital archiving
  • photogrammetry and 3D capture
  • AI-assisted analysis
  • systems that track objects or behavior over time

These problems are messy.

Which is part of what makes them interesting.

A Continuing Exploration

The posts I’ve been writing are not meant to be definitive.

They’re part of an ongoing exploration.

A way to think through problems in public.

And occasionally, a way to use a slightly unusual example — like a pile of rocks — to make broader ideas easier to see.

If You’re Interested

If any of this resonates, you might find these useful:

The Backyard Quarry Series

A systems-focused look at modeling and working with physical objects starting with Turning Rocks Into Data.

MCP and AI Systems

A technical exploration of building agent-based systems and data pipelines. I’d suggest starting with The End of Glue Code: Why MCP is the USB-C Moment for AI Systems.

More to come.

And if nothing else, it turns out that even a backyard can be a good place to think about system design.

Facebooktwitterredditlinkedinmail

The Backyard Quarry: Turning Rocks Into Data

Another round of tech layoffs rolled through the industry recently, and I was one of the people caught in it.

If you’ve worked in tech for any length of time, you know the routine that follows. Update the résumé. Reach out to contacts. Scroll job boards. Try to figure out which technologies the market is currently excited about and which ones have quietly drifted into irrelevance.

After a few days of that cycle, I found myself spending more time outside than in front of a laptop.

One afternoon, walking around the yard, I noticed something interesting.

My backyard contains a surprisingly large dataset.

Rocks.

Pile of Rocks
Sample of the rocks from the Backyard Quarry used for the dataset.

Lots of rocks.

Some are the size of peas. Others are roughly the size of a car engine. A few fall somewhere in the unsettling range between “wheelbarrow recommended” and “this probably requires heavy machinery.”

Naturally, I had the same thought many people eventually have when staring at a large pile of rocks:

I could probably sell these.

A small stand near the road. A few piles sorted by size. Maybe a sign that says “Landscaping Rock.” It’s not exactly a venture-backed startup, but stranger side businesses have existed.

Unfortunately, engineers have a well-known weakness.

We rarely do things the simple way.

If I was going to sell rocks, I wasn’t just going to pile them on a table.

I was going to build a system.

The Dataset

The moment you start thinking about the rocks as inventory, a familiar set of questions appears.

How many rocks are there?

What kinds?

Which ones are small decorative stones and which ones fall firmly into what I’ve started calling Engine Block Class?

Like many real-world datasets, this one has significant variability.

Some objects are a few grams. Others weigh enough to require careful lifting technique and a brief internal conversation about life choices.

At a glance, the dataset looks chaotic. But underneath the chaos are patterns.

Different sizes. Different shapes. Different colors. Different geological types. Some rocks are smooth river stones. Others are jagged fragments that look like they escaped from a small landslide.

If you squint a little, you start to see the outlines of something familiar to anyone who works with data systems.

A collection of physical objects that could be represented as structured records.

The Engineer’s Curse

In theory, selling rocks is simple.

  • Step one: collect rocks.
  • Step two: put them in a pile.
  • Step three: wait for someone to stop their car and decide they want landscaping material.

But once you start thinking about it from an engineering perspective, the questions multiply.

Should each rock have an identifier?

Should there be photographs?

Should the system track weight or dimensions?

What about classification?

It’s probably useful to distinguish between Pebble Class rocks and Wheelbarrow Class rocks.

And what about the really large ones — the ones that are clearly in the Engine Block Class, which itself appears to span everything from motorcycle engine scale to something closer to a semi-truck.

Once you start thinking about these questions, the simple rock pile begins to look like something else entirely.

A catalog.

A dataset.

A system waiting to happen.

From Rocks to Records

What if every rock had a record?

Something simple at first.

An identifier. A few attributes. Maybe a photo.

Conceptually, it might look like this:

rock_id
weight
dimensions
color
rock_type
location_found
status

Each rock in the yard becomes a digital object — a structured record representing something in the physical world.

In other words, each rock now has a digital twin.

That might sound slightly ridiculous in the context of landscaping stones, but the idea is surprisingly powerful.

Across many industries, organizations are trying to solve exactly this problem: how to connect messy physical reality with structured digital systems.

Manufacturers track machine parts.

Museums catalog artifacts.

Farmers track crops.

Logistics companies track inventory moving through warehouses.

In each case, the challenge is similar.

A physical object exists somewhere in the world.

We want to represent it in a way that software systems can understand.

The Backyard Quarry

At this point the rock pile had acquired a new name.

The Backyard Quarry.

Partly as a joke, and partly because it captured the spirit of the project. What started as a casual observation had turned into a small experiment in data modeling, object cataloging, and system design.

The dataset might be small.

The objects might be rocks.

But the underlying questions are surprisingly rich.

How do you represent physical objects in software?

How do you capture information about them?

How do you search and organize the resulting data?

And how do these systems scale when the number of objects grows from a few dozen to thousands — or millions?

What Comes Next

Over the next few posts in this series, I’m going to explore those questions by building a small system around the Backyard Quarry.

We’ll look at things like:

  • designing a schema for physical objects
  • capturing images and measurements
  • generating 3D models using photogrammetry
  • building ingestion pipelines
  • indexing and searching the dataset

All starting from a simple collection of rocks.

The world has no shortage of complicated engineering problems.

Sometimes the best place to explore them is somewhere simpler.

Like a pile of rocks in the backyard.

And if you happen to need a carefully documented specimen from the Backyard Quarry, inventory is currently available.

Shipping, however, may exceed the value of the rock itself.

Facebooktwitterredditlinkedinmail