Category: AI

What I’ve Been Building: Systems, AI, and Real-World Data

Over the past several weeks, I’ve been spending a lot of time thinking about systems.

Some of that thinking has taken the form of writing.

If you’ve come across any of my recent posts, they might seem like they cover very different topics:

  • cataloging rocks in a backyard
  • building AI systems using MCP
  • working with documents, images, and real-world data

At first glance, they don’t appear to have much in common.

But they’re all exploring the same underlying idea.

The Common Thread

Across all of these posts, the focus has been on a specific kind of problem:

How do we turn messy, real-world inputs into structured, usable systems?

That problem shows up in many different forms.

Sometimes the input is physical:

  • objects
  • artifacts
  • environments

Sometimes it’s digital:

  • documents
  • images
  • logs

Sometimes it’s dynamic:

  • motion
  • behavior
  • sensor data

But the challenge is the same.

The input is unstructured.

The system needs structure.

The Backyard Quarry

One way I explored this idea was through a small project I called the Backyard Quarry.

It started with a simple observation:

There are a lot of rocks in the yard.

From there, the problem evolved into something more interesting:

  • how to represent physical objects as data
  • how to capture images and measurements
  • how to build pipelines around that data
  • how to search and organize it
  • how to think about digital twins

What began as a small experiment became a way to explore system design in a constrained, tangible setting.

MCP and AI Systems

In parallel, I’ve been writing about building AI systems using MCP.

On the surface, this looks very different.

Instead of rocks, the inputs are:

  • documents
  • APIs
  • models
  • agent workflows

But the structure is familiar.

  • inputs are ingested
  • processed
  • transformed
  • routed
  • used by applications

The system still needs to handle:

  • variability
  • scale
  • imperfect data
  • orchestration

Different inputs.

Same patterns.

From Objects to Systems

One of the more useful realizations in working through these ideas is this:

The problem is rarely about the individual object.
It’s about the system that handles many objects over time.

Whether the object is:

  • a rock
  • a document
  • a sensor reading

The questions become:

  • how is it represented?
  • how does it enter the system?
  • how is it transformed?
  • how is it stored?
  • how is it retrieved?

These are system-level questions.

A Shared Architecture

Across these different domains, a common architecture begins to emerge.

Diagram showing how raw inputs are captured, processed, structured, indexed, and used by applications in a data system.
A common pattern for transforming real-world inputs into usable systems.

The labels change depending on the domain.

But the structure remains consistent.

Why This Matters

Understanding this pattern makes it easier to approach new problems.

Instead of starting from scratch each time, you can ask:

  • Where does the data come from?
  • How does it enter the system?
  • What transformations are required?
  • How will it be used?

This reduces complexity.

It also makes systems more predictable.

What I’m Interested In

Going forward, I’m particularly interested in systems that sit at the boundary between:

  • the physical world and digital systems
  • unstructured inputs and structured data
  • human workflows and automated processes

That includes areas like:

  • digital archiving
  • photogrammetry and 3D capture
  • AI-assisted analysis
  • systems that track objects or behavior over time

These problems are messy.

Which is part of what makes them interesting.

A Continuing Exploration

The posts I’ve been writing are not meant to be definitive.

They’re part of an ongoing exploration.

A way to think through problems in public.

And occasionally, a way to use a slightly unusual example — like a pile of rocks — to make broader ideas easier to see.

If You’re Interested

If any of this resonates, you might find these useful:

The Backyard Quarry Series

A systems-focused look at modeling and working with physical objects starting with Turning Rocks Into Data.

MCP and AI Systems

A technical exploration of building agent-based systems and data pipelines. I’d suggest starting with The End of Glue Code: Why MCP is the USB-C Moment for AI Systems.

More to come.

And if nothing else, it turns out that even a backyard can be a good place to think about system design.

Facebooktwitterredditlinkedinmail

The Accountant: Optimizing AI Costs with Semantic Routing

We’ve solved the Reliability problem with The Judge. We have a system that can scientifically prove whether our Forensic Team is accurate. But there’s a new problem that keeps Directors and CFOs up at night: Sustainability.

In an enterprise environment, using a massive, high-reasoning model (like Claude 3.5 or GPT-4o) for every single bibliography lookup is a “Cognitive Budget” disaster. It’s like hiring a Senior Architect to fix a broken link.

Today, we introduce The Accountant: A Semantic Router that classifies task complexity and routes requests to the cheapest model capable of passing the Judge’s rubric.

1. The Concept of “Tiered Intelligence”

Not all forensic tasks require the same level of “gray matter.” To scale effectively, we must categorize our workload:

  • LEVEL 1 (Operational): “Find the standard page count for the 1925 edition of Gatsby.” This is a lookup and retrieval task. Local SLMs (Small Language Models) like Phi-4 or Llama 3.2 excel here.
  • LEVEL 2 (Forensic): “Compare the binding grain and typography inconsistencies between two suspected forgeries.” This requires high-dimensional analysis and deep reasoning. This is a job for the Cloud.
Architectural diagram of a Semantic Router called The Accountant. A user request enters the router, which classifies it into Level 1 (Simple/Metadata) or Level 2 (Complex Forensic). Level 1 is routed to a local Tier 1 SLM like Phi-4 or Llama 3.2, while Level 2 is routed to a Tier 2 Frontier Cloud model like Claude 3.5. Both paths converge to produce a final Audit Report.
The Semantic Router Architecture—Implementing Tiered Intelligence to optimize cognitive budget and reduce inference costs.

2. Implementing the Router (The Gatekeeper Pattern)

We’ve added router.py to our repository. The logic acts as a gatekeeper.
1. Classification: A lightweight model (the Accountant) reviews the user’s query against our config/prompts.yaml.
2. Economic Decision: If the query is “Level 1”, we trigger the ollama provider. If it’s “Level 2,” we escalate to the anthropic provider.

# The Accountant's Decision Engine in router.py
level = await classify_query(query)
provider = get_provider_for_level(level)

if level == "LEVEL_1":
    print("Accountant Decision: LEVEL_1 - Routing to Local SLM to save budget")
else:
    print("Accountant Decision: LEVEL_2 - Routing to High-Reasoning Cloud Model")

By defaulting to LEVEL_2 if classification fails, we ensure that we never sacrifice accuracy for cost – we only save money when we are certain the tasks are simple.

3. Projecting the ROI with The Judge

While we built the Accountant (the router), we haven’t yet run a full-scale economic audit in this repository. However, the architecture is designed to scientifically measure this trade-off using the Judge Agent (from our last post).

In an enterprise environment, a Director would use this framework to benchmark a representative sample of historical queries. A typical analysis for tiered intelligence systems shows that the vast majority of “forensic” requests are actually simple metadata lookups. By routing those to a local SLM (Phi-4 or Llama 3.2), we can achieve comparable reliability scores to a frontier cloud model while zeroing out the marginal cost of those specific tokens.

The Theoretical Savings (100k Calls/Month):

  • Current Cost (Frontier Cloud for 100% of tasks): ~$7,600/month
  • Projected Cost (90/10 Routed Split): ~$1,800/month
  • Total Savings: ~76% reduction in inference costs.
Task Category Estimated Volume “Status Quo” Cost (Frontier Cloud) “Routed” Cost (Accountant/SLM)
Level 1 (Standard Lookup/Formatting) 90% (90k calls) ~$4,500 ~$0 (Local/Self-Hosted)
Level 2 (Deep Forensic Analysis) 10% (10k calls) ~$3,100 ~$1,800*
Total Cognitive Budget 100% ~$7,600 ~$1,800

* Note: Level 2 “Routed” costs are lower here because the Accountant ensures only the most complex 10% of tokens hit the high-cost provider, whereas the “Status Quo” assumes a higher average cost across all 100k calls due to the lack of optimization.

Cognitive Budgeting Insights

As a Director, the responsibility is to build Sustainable Intelligence. If 80% of an AI workload can be moved to local infrastructure or cheaper “Flash” models without dropping our reliability score, I’m not just a developer—I’m a profit center. Semantic routing allows us to scale AI horizontally without the cloud bill scaling vertically.

🛠️ Step into the Clean-Room

The Accountant logic is now live in the repository. You can test the routing logic yourself by running the local orchestrator with the --use-accountant flag.

Explore the Code: MCP Forensic Analyzer on GitHub

(If this architecture helps your team justify their AI spend, consider dropping a ⭐ on the repo!)

The Production-Grade AI Series

  • Post 1: The Judge Agent: Who Audits the Auditors? (Reliability)
  • Post 2: The Accountant: Optimizing AI Costs with Semantic Routing (Sustainability) – You’re Here
  • Post 3: The Guardian: Human-in-the-Loop Governance (Safety) – Coming Soon

Looking for the foundation? Check out my previous series: The Zero-Glue AI Mesh with MCP.

Facebooktwitterredditlinkedinmail