Mapping the Social Graph

In our previous post, we built the Knowledge Archive, a durable, atomic vault for historical data. We moved from “strings to things” by adopting the Schema.org vocabulary. But even with a vault full of people, the Scribe was still seeing individuals in isolation.

History isn’t just a list of names; it’s a web of Relationships. Today, we move from the Archive to the Social Graph.

Beyond the Nuclear Family

The 1880 Census is a social map. In a single dwelling in Salem, Oregon, you might find a Head of Household, his wife, four children, two boarders (often young laborers), and a live-in servant.

To capture this, the Digital Scribe needs to move beyond simple genealogy. We’ve engineered a Relational Linker that recognizes the “connective tissue” of a household:

  • Nuclear Ties: Mapping “Wife” and “Husband” to schema:spouse and “Son/Daughter” to schema:parent.
  • Extended Ties: Mapping “Boarder,” “Servant,” and “Cook” to a memberOfHousehold relationship.

The scribe uses a specialized ‘Idempotent Appender.’ This function is a surgical tool: it checks if a relationship already exists before adding it. If it finds a bare string ID from an older version of the archive, it ‘promotes’ it to a structured dictionary, ensuring the archive’s schema remains perfectly consistent over time.

# The Idempotent Relation Appender
def _add_to_relation(entity: dict, property_name: str, value: dict) -> bool:
    target_id = value.get("@id")
    existing = entity.get(property_name)

    # Promotion logic: Convert bare strings to structured dictionaries
    if isinstance(existing, str):
        if existing == target_id: return False
        entity[property_name] = [{"@id": existing}, value]
        return True

    # ... handles lists and deduplication to prevent redundant data ...
graph LR
    %% --- Central Entity ---
    Head[Head of Household]

    %% --- Nuclear Family ---
    subgraph Nuclear ["Nuclear Unit"]
        Spouse[Spouse: Wife or Husband]
        Child[Son or Daughter]
    end

    %% --- Extended Household ---
    subgraph Extended ["Extended Household"]
        Boarder[Boarder]
        Servant[Servant]
    end

    %% --- Relationship Edges ---
    %% Symmetric Spouse Link
    Head ---|spouse| Spouse
    Spouse ---|spouse| Head

    %% Parent/Child Links
    Child -->|parent| Head
    Head -.->|knows| Child

    %% Extended links
    Boarder -->|memberOfHousehold| Head
    Servant -->|memberOfHousehold| Head

    Head -.->|knows| Boarder
    Head -.->|knows| Servant

    %% --- Styles ---
    style Head fill:#f9f,stroke:#333,stroke-width:2px
    style Spouse fill:#bbf,stroke:#333
    style Child fill:#bbf,stroke:#333
    style Boarder fill:#eee,stroke:#999
    style Servant fill:#eee,stroke:#999

This graph highlights the multi-modal nature of historical memory. By using solid lines for core family ties and dashed lines for the “extended” household (Boarders and Servants), we maintain the distinction between biological lineage and social proximity. Note the bidirectional “spouse” arrows; in our graph, no one is a silent attribute—everyone is a first-class node capable of pointing back to their connections.

The Engineering of Symmetry

In a true Knowledge Graph, relationships must be Symmetric. If the Scribe identifies a “Wife,” it shouldn’t just point her to the Head of Household; the Head must also point back to her.

We implemented a Symmetric Linking Pipeline that ensures the graph is balanced. When the Scribe “forges” a link, it updates both entities simultaneously within a single atomic transaction.

To build a graph that actually works, the Scribe must be unbiased. It doesn’t just link a ‘Wife’ to a ‘Head’; it links them both as equals in a spouse relationship. This ensures that no matter which person the AI ‘looks’ at first, it can find the other.

# The Scribe's Symmetric Linking Pipeline
if rel_lower in ("wife", "husband"):
    # Mirroring the census reality: Both partners are linked symmetrically
    spouse_link = {"@id": head_id, "relationshipDescription": rel_raw}
    spouse_back = {"@id": member_id, "relationshipDescription": rel_raw}

    # Update Member -> Head
    if _add_to_relation(entity, "spouse", spouse_link):
        links_created += 1
    # Update Head -> Member (Symmetry)
    if _add_to_relation(head, "spouse", spouse_back):
        links_created += 1

Mapping the Block: The Dwelling as a Container

One of the most powerful tools we’ve added is search_by_dwelling. By treating the physical house as a container, the Scribe can now “Map the Block.” We can ask the Scribe to show us everyone living at Dwelling #10, revealing multi-family boarding houses and the complex social hierarchies within.

By combining this search with our Dry Run feature, the Scribe can “imagine” the social links of an entire neighborhood before committing them to the permanent archive.

# Mapping a multi-family dwelling via the MCP tool
dwelling_data = mcp.call_tool("search_by_dwelling", {"dwelling_number": 10})

# Logic: Dwelling 10 contains everyone in the building
# including Boarders and different Family Numbers.
# Output Count: 11 residents
# - Family 12: The Smith Family
# - Family 13: The Miller Family
# - Unaffiliated: John Doe (Boarder)

Because search_by_dwelling returns a structured list of all residents, the agent can iterate through multiple family units in a single pass, applying the graph-linking logic to the entire physical structure at once.

graph TD
    %% --- The Physical Container (Dwelling 10) ---
    subgraph Building ["Dwelling 10: The Physical Building"]

        %% --- Household A ---
        subgraph Family12 ["Family Number 12"]
            A1(Head: Farmer)
            A2(Wife)
            A3(Son)
        end

        %% --- Household B ---
        subgraph Family13 ["Family Number 13"]
            B1(Head: Blacksmith)
            B2(Son)
            B3(Daughter)
        end

        %% --- Unaffiliated ---
        C1[Servant]
        C2[Boarder]
    end

    %% --- Inter-Dwelling Links ---
    A1 <--> B1
    A1 <--> C1
    B1 <--> C2

    %% --- Styles ---
    style Building fill:#eef,stroke:#333,stroke-width:2px
    style Family12 fill:#f9f,stroke:#333
    style Family13 fill:#bbf,stroke:#333
    style A1,B1 fill:#f9f,stroke:#333,stroke-width:1px

Viewing history through the “Dwelling” lens reveals a different kind of truth. By treating the physical building as a parent container, the Scribe can group disparate family units (like Family 12 and 13 above) who shared the same roof. This “Mapping the Block” strategy allows an agent to infer social influence—how a Boarder’s trade might influence the children of the family they live with, or how neighborhoods clustered by occupation.

Why the Graph is the Future

By building a Social Graph, we’ve given the Digital Scribe a form of “Inference.” It no longer just knows who people are; it understands how they belong.

This is the final foundation stone for the Digital Scribe. We have mastered Capture, Persistence, and Connectivity.

What’s Next?

The Digital Scribe is now a fully realized Knowledge Graph. We have mastered Capture, Persistence, and Connectivity. But a Sovereign system shouldn’t just live in the past. In our next entry, we’re going to take a ‘Brain Break’ to look at how this exact same architecture can be handed over to a modern challenge: helping a farmer pivot their harvest when the market shifts.

The 1880 Archive was the training ground; the 2026 Vineyard is the mission.

Facebooktwitterredditlinkedinmail

Engineering the Knowledge Archive

In our last post, we introduced the Digital Scribe, an AI architecture designed to capture the “unstructured nightmare” of historical records. We showed how the Scribe uses the Model Context Protocol (MCP) to transcribe 19th-century cursive and resolve the cryptic “ditto marks” of the past.

But transcription is only half the battle. If the Scribe forgets what it read the moment the session ends, we haven’t built a system; we’ve just built a fancy typewriter.

Today, we go deeper into the Scribe’s Memory.

Memory is an Engineering Discipline

As I’ve written before in Engineering Agent Memory, AI agents are often “stateless by default.” They live in the moment, relying on a flat conversation transcript that grows until it hits a token limit.

For the Digital Scribe, that is unacceptable. To digitize the 1880 Census of Salem, Oregon, we need Semantic Memory, a way to store, index, and retrieve knowledge intentionally.

The Architecture of Persistence: JSON-LD

We didn’t just want a text file; we wanted a Sovereign Archive. We chose JSON-LD (JSON for Linked Data) aligned with Schema.org standards. This transforms a census row into a “Thing, not a string.”

To achieve this, we don’t just dump JSON; we map our historical model to the Schema.org Person vocabulary. This ensures that a ‘Scribe’ in 2026 and a researcher in 2050 can both understand that a ‘birthplace’ string is actually a Schema.org/Place entity.

# Mapping the Census to the Global Schema
def _record_to_jsonld_entity(record: Census1880Record, entity_id: str | None = None) -> dict:
    given, family = _parse_historical_name(record.name)
    return {
        "@context": "https://schema.org/",
        "@type": "Person",
        "@id": entity_id or f"urn:uuid:{uuid.uuid4()}",
        "givenName": given,
        "familyName": family,
        "hasOccupation": {"@type": "Occupation", "name": record.occupation},
        "birthPlace": {"@type": "Place", "name": record.birthplace},
        "censusFamilyNumber": record.family_number,
        "censusDwellingNumber": record.dwelling_number,
    }

Technical Deep Dive: Parsing Historical Names

In 1880, names weren’t always “First Last.” We built a robust parser to handle “Surname, Given Name” formats and multi-word surnames. Without this, our “Semantic Memory” would be fractured by simple formatting variances.

Input String givenName familyName
“Smith, John” “John” “Smith”
“Mary Ann Jones” “Mary Ann” “Jones”
“John Smith” “John” “Smith”

When the Scribe identifies “John Smith” in a ledger, it doesn’t just save a name. It creates a Schema.org/Person entity, complete with a unique urn:uuid: and structured links to his occupation and birthplace.

Atomic Ingestion: Protecting the History

Because we are building “Sovereign Infrastructure,” the integrity of the data is paramount. We implemented an Atomic Write Pattern to ensure the archive is never corrupted.

  1. Thread-Safety: A global lock ensures that multiple “Scribe” agents don’t collide when writing to the same archive.
  2. Write-Ahead Strategy: The system writes to a temporary file and uses os.replace only after the data is verified.
  3. Durability: We use os.fsync to ensure the data is physically flushed to the disk, protecting against power loss or OS crashes.

By using a write-to-temp pattern followed by an os.fsync, we ensure that the data is physically committed to the platter before we ever swap it into the main archive. This prevents ‘half-written’ files if the power cuts or the process crashes.

# The "Sovereign" Atomic Save
def _save_graph(self, entities: list[dict]) -> None:
    tmp_path = self._path.with_suffix(self._path.suffix + ".tmp")
    replaced = False
    try:
        with open(tmp_path, "w", encoding="utf-8") as f:
            json.dump(entities, f, indent=2, ensure_ascii=False)
            f.write("\n")
            f.flush()
            os.fsync(f.fileno()) # Force the OS to flush to disk
        os.replace(tmp_path, self._path) # Atomic swap
        replaced = True
    finally:
        if not replaced and tmp_path.exists():
            tmp_path.unlink() # Cleanup if we failed

The Recall: Deduplication and Entity Intelligence

The true power of the Scribe’s memory is revealed during Ingestion. If we attempt to capture the same person twice, the Scribe doesn’t just blindly append the data. It performs a Deduplication Check.

By hashing the record’s “DNA” (Name, Dwelling, and Family Number), the Scribe recognizes “John Smith” from a previous run and skips the ingestion, returning a duplicate_skipped status.

Deduplication is the ultimate test of a Scribe’s integrity. We define a unique fingerprint for each life, e.g. a combination of their Name, Dwelling, and Family Number. If the Scribe sees this ‘DNA’ again, it refuses to create a duplicate, maintaining a clean, high-fidelity archive.

# The Knowledge Stewardship Guard
for e in entities:
    if (
        (e.get("givenName") or "") == given
        and (e.get("familyName") or "") == family
        and e.get("censusDwellingNumber") == record.dwelling_number
        and e.get("censusFamilyNumber") == record.family_number
    ):
        # Already exists—identify it and move on
        existing_id = e.get("@id") or f"{LEGACY_ID_PREFIX}{_content_hash(e)}"
        return (existing_id, False)

A detailed architectural diagram of the Digital Scribe's Semantic Memory layer. It shows the flow from structured JSON through name parsing and entity fingerprinting, into a persistent JSON-LD archive protected by threading locks, corruption guards, and fsync durability.

Why This Matters: Building the Graph

By engineering a persistent, semantic memory, we’ve given the Scribe the ability to recall context across time.

In our next post, we will use this foundation to move from individual residents to The Knowledge Graph. We will begin linking families, neighborhoods, and migration patterns—turning a static archive into a living map of the past.

The Digital Scribe isn’t just reading history anymore. It’s remembering it.

Facebooktwitterredditlinkedinmail