{"id":1418,"date":"2026-06-18T04:29:00","date_gmt":"2026-06-18T11:29:00","guid":{"rendered":"https:\/\/www.kenwalger.com\/blog\/?p=1418"},"modified":"2026-04-23T07:58:01","modified_gmt":"2026-04-23T14:58:01","slug":"engineering-ai-agent-memory-json-ld","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/ai\/engineering-ai-agent-memory-json-ld\/","title":{"rendered":"Engineering the Knowledge Archive"},"content":{"rendered":"

In our last post<\/a>, we introduced the Digital Scribe<\/strong>, an AI architecture designed to capture the “unstructured nightmare” of historical records. We showed how the Scribe uses the Model Context Protocol (MCP)<\/a> to transcribe 19th-century cursive and resolve the cryptic “ditto marks” of the past.<\/p>\n

But transcription is only half the battle. If the Scribe forgets what it read the moment the session ends, we haven’t built a system; we\u2019ve just built a fancy typewriter.<\/p>\n

Today, we go deeper into the Scribe\u2019s Memory<\/strong>.<\/p>\n

Memory is an Engineering Discipline<\/h2>\n

As I\u2019ve written before in Engineering Agent Memory<\/a>, AI agents are often “stateless by default.” They live in the moment, relying on a flat conversation transcript that grows until it hits a token limit.<\/p>\n

For the Digital Scribe, that is unacceptable. To digitize the 1880 Census of Salem, Oregon, we need Semantic Memory<\/strong>, a way to store, index, and retrieve knowledge intentionally.<\/p>\n

The Architecture of Persistence: JSON-LD<\/h2>\n

We didn’t just want a text file; we wanted a Sovereign Archive<\/em>. We chose JSON-LD (JSON for Linked Data)<\/a> aligned with Schema.org<\/a> standards. This transforms a census row into a “Thing, not a string.”<\/p>\n

To achieve this, we don’t just dump JSON; we map our historical model to the Schema.org Person<\/code> vocabulary. This ensures that a ‘Scribe’ in 2026 and a researcher in 2050 can both understand that a ‘birthplace’ string is actually a Schema.org\/Place<\/code> entity.<\/p>\n

# Mapping the Census to the Global Schema\ndef _record_to_jsonld_entity(record: Census1880Record, entity_id: str | None = None) -> dict:\n    given, family = _parse_historical_name(record.name)\n    return {\n        \"@context\": \"https:\/\/schema.org\/\",\n        \"@type\": \"Person\",\n        \"@id\": entity_id or f\"urn:uuid:{uuid.uuid4()}\",\n        \"givenName\": given,\n        \"familyName\": family,\n        \"hasOccupation\": {\"@type\": \"Occupation\", \"name\": record.occupation},\n        \"birthPlace\": {\"@type\": \"Place\", \"name\": record.birthplace},\n        \"censusFamilyNumber\": record.family_number,\n        \"censusDwellingNumber\": record.dwelling_number,\n    }\n<\/code><\/pre>\n
Technical Deep Dive: Parsing Historical Names<\/p><\/blockquote>\n
In 1880, names weren’t always “First Last.” We built a robust parser to handle “Surname, Given Name” formats and multi-word surnames. Without this, our “Semantic Memory” would be fractured by simple formatting variances.<\/p>\n\n\n\n\n\n\n\n
Input String<\/th>\ngivenName<\/code><\/th>\nfamilyName<\/code><\/th>\n<\/tr>\n<\/thead>\n
“Smith, John”<\/td>\n“John”<\/td>\n“Smith”<\/td>\n<\/tr>\n
“Mary Ann Jones”<\/td>\n“Mary Ann”<\/td>\n“Jones”<\/td>\n<\/tr>\n
“John Smith”<\/td>\n“John”<\/td>\n“Smith”<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
When the Scribe identifies “John Smith” in a ledger, it doesn’t just save a name. It creates a Schema.org\/Person<\/code> entity, complete with a unique urn:uuid:<\/code> and structured links to his occupation and birthplace.<\/p>\n
Atomic Ingestion: Protecting the History<\/h3>\n
Because we are building “Sovereign Infrastructure,” the integrity of the data is paramount. We implemented an Atomic Write Pattern<\/strong> to ensure the archive is never corrupted.<\/p>\n
    \n
  1. Thread-Safety:<\/strong> A global lock ensures that multiple “Scribe” agents don’t collide when writing to the same archive.<\/li>\n
  2. Write-Ahead Strategy:<\/strong> The system writes to a temporary file and uses os.replace<\/code> only after the data is verified.<\/li>\n
  3. Durability:<\/strong> We use os.fsync<\/code> to ensure the data is physically flushed to the disk, protecting against power loss or OS crashes.<\/li>\n<\/ol>\n
    By using a write-to-temp pattern followed by an os.fsync<\/code>, we ensure that the data is physically committed to the platter before we ever swap it into the main archive. This prevents ‘half-written’ files if the power cuts or the process crashes.<\/p>\n
    # The \"Sovereign\" Atomic Save\ndef _save_graph(self, entities: list[dict]) -> None:\n    tmp_path = self._path.with_suffix(self._path.suffix + \".tmp\")\n    replaced = False\n    try:\n        with open(tmp_path, \"w\", encoding=\"utf-8\") as f:\n            json.dump(entities, f, indent=2, ensure_ascii=False)\n            f.write(\"\\n\")\n            f.flush()\n            os.fsync(f.fileno()) # Force the OS to flush to disk\n        os.replace(tmp_path, self._path) # Atomic swap\n        replaced = True\n    finally:\n        if not replaced and tmp_path.exists():\n            tmp_path.unlink() # Cleanup if we failed\n<\/code><\/pre>\n
    The Recall: Deduplication and Entity Intelligence<\/h2>\n
    The true power of the Scribe\u2019s memory is revealed during Ingestion. If we attempt to capture the same person twice, the Scribe doesn’t just blindly append the data. It performs a Deduplication Check<\/strong>.<\/p>\n
    By hashing the record’s “DNA” (Name, Dwelling, and Family Number), the Scribe recognizes “John Smith” from a previous run and skips the ingestion, returning a duplicate_skipped<\/code> status.<\/p>\n
    Deduplication is the ultimate test of a Scribe’s integrity. We define a unique fingerprint for each life, e.g. a combination of their Name, Dwelling, and Family Number. If the Scribe sees this ‘DNA’ again, it refuses to create a duplicate, maintaining a clean, high-fidelity archive.<\/p>\n
    # The Knowledge Stewardship Guard\nfor e in entities:\n    if (\n        (e.get(\"givenName\") or \"\") == given\n        and (e.get(\"familyName\") or \"\") == family\n        and e.get(\"censusDwellingNumber\") == record.dwelling_number\n        and e.get(\"censusFamilyNumber\") == record.family_number\n    ):\n        # Already exists\u2014identify it and move on\n        existing_id = e.get(\"@id\") or f\"{LEGACY_ID_PREFIX}{_content_hash(e)}\"\n        return (existing_id, False)\n<\/code><\/pre>\n
    \"A<\/p>\n
    Why This Matters: Building the Graph<\/h2>\n
    By engineering a persistent, semantic memory, we\u2019ve given the Scribe the ability to recall context across time.<\/p>\n
    In our next post, we will use this foundation to move from individual residents to The Knowledge Graph. We will begin linking families, neighborhoods, and migration patterns\u2014turning a static archive into a living map of the past.<\/p>\n
    The Digital Scribe isn’t just reading history anymore. It\u2019s remembering it.<\/p>\n\"Facebook\"<\/a>\"twitter\"<\/a>\"reddit\"<\/a>\"linkedin\"<\/a>\"mail\"<\/a>","protected":false},"excerpt":{"rendered":"
    In our last post, we introduced the Digital Scribe, an AI architecture designed to capture the “unstructured nightmare” of historical records. We showed how the Scribe uses the Model Context Protocol (MCP) to transcribe 19th-century cursive and resolve the cryptic “ditto marks” of the past. But transcription is only half the battle. If the Scribe … Continue reading “Engineering the Knowledge Archive”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":1505,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pmpro_default_level":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1669,1670],"tags":[1774,1668,1736,1773,78,1725],"yst_prominent_words":[99,768],"class_list":["post-1418","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-mcp","tag-agentic-memory","tag-ai","tag-data-engineering","tag-json-ld","tag-python","tag-sovereign-ai","pmpro-has-access"],"jetpack_featured_media_url":"https:\/\/www.kenwalger.com\/blog\/wp-content\/uploads\/2026\/04\/blog-of-ken-w.-alger-69ea335ce001a.png","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8lx70-mS","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1418","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/comments?post=1418"}],"version-history":[{"count":5,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1418\/revisions"}],"predecessor-version":[{"id":1424,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/posts\/1418\/revisions\/1424"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media\/1505"}],"wp:attachment":[{"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/media?parent=1418"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/categories?post=1418"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/tags?post=1418"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.kenwalger.com\/blog\/wp-json\/wp\/v2\/yst_prominent_words?post=1418"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}