Metadata served via OKF

We had a few hundred governed concepts to store. Definitions, the rules for using them, the defaults that apply when no rule fits. Postgres was already running next to everything else, so that’s where they went. Tables, migrations, an API on top. Completely standard, and I’d do it again for most things.

It fell apart the first time someone who actually owns the definitions wanted to change one.

The problem isn’t the database #

It’s what a database does to a review process.

Changing one sentence of guidance meant a migration. A script, a review of the script, a deploy. That’s a lot of machinery for a typo, and the predictable result is that typos don’t get fixed. You learn to live with the wrong word.

And the review itself is strange, because there’s nothing to look at. Nobody can review an UPDATE. The thing you want a second pair of eyes on is these words became those words, and that artefact only exists in someone’s head. What gets reviewed is a diff of SQL, which is a proxy for the change at best.

Then somebody asked what a definition said in March and who signed it off. We didn’t have a great answer. You can build that, obviously. Audit columns, temporal tables, an events table. But you have to decide to build it up front, and what you’re building is a worse version of something already sitting on your laptop.

The one that finally did it: every time the source grew a column, we needed a migration, a model change, a serialiser change and an API change before anyone could see the new field. So coverage stopped wherever it was convenient to stop. The owners of the data couldn’t add anything themselves, couldn’t read it without a UI, and couldn’t fix a sentence without filing a ticket. Governance that depends on engineering availability isn’t governance. It turns back into a spreadsheet someone emails around, which is where the whole thing started.

So we moved it out. The split we settled on, roughly: operational data stays in Postgres, documentary knowledge goes in files. Per-project rows, ingest state, pipeline history, all still in the database. Concept definitions, rules, guidance, all files in git.

Why OKF and not just, you know, markdown #

“Put it in markdown” isn’t a design. I know what a folder of ad-hoc .md files looks like after six months of two people adding fields however they feel. A wiki is markdown too and is fine for humans and useless for anything that needs to parse it.

What I wanted was the smallest contract a parser could rely on that a human could also completely ignore. OKF is that, and the reason I picked it is mostly that it barely asks for anything.

A bundle is a directory of UTF-8 markdown files, one per concept. Each file is YAML frontmatter and then a body. For it to be conformant, every non-reserved file needs parseable frontmatter with a non-empty type key. That’s the only field that’s ever required. The reserved files (index.md, log.md) have a defined shape, and the root index can carry a version key and nothing else. That’s the whole spec surface I had to care about.

Four things about it lined up with the four things that were hurting:

The unit of change is a file. One concept, one file, one diff, one PR. Review and history stopped being things I had to design.

Frontmatter is open. Producers can add any keys they want, and consumers aren’t allowed to reject a bundle over keys they don’t recognise. They’re also supposed to preserve unknown keys when they round-trip a file. Our concept files carry something like thirty keys now, most of them ours, and they’re still conformant. Adding a field from the source is a converter change and nothing else.

Humans and machines read the same bytes. Frontmatter holds the single-valued attributes that code filters on. The body holds the tables and the prose and the edge cases. There’s no export step, so there’s no second copy to drift.

There’s nothing to adopt. No server, no schema registry, no client library. If you can list a directory and parse YAML you’re a consumer. That’s why we ended up with four consumers instead of four integration projects.

That open-frontmatter rule has a catch I’d underline for anyone starting. If you build a UI that edits these files, it has to round-trip keys it doesn’t understand. Ours didn’t at first. The form serialised the fields it had widgets for and quietly dropped everything else on save, which we found the way you’d expect to find it. Build the passthrough before the first edit ships.

Generating the files #

Nobody writes the markdown by hand. The source is still a workbook, and a converter walks it and writes files. Load the sheets, resolve each concept across them, build frontmatter, build body sections, write, validate, regenerate the static artefacts.

The only genuinely interesting part is precedence, because plenty of fields show up on more than one sheet with different values. Every field gets an ordered list of sheets and we take the first non-empty one. The order isn’t arbitrary: the cleaned machine-generated sheet beats the one an analyst wrote in prose, because one is a schema and the other is a description of a schema.

Four things that cost me time, in case they save you some:

Don’t join on display names. The same concept is spelled differently on different sheets, and one row has a straightforward misspelling in the name. A name join doesn’t error, it just quietly drops the row. Join on a normalised file stem instead.

Check for fields spread over two columns. One piece of guidance lived in two adjacent columns, and reading only the first silently lost half the values. Nothing in the pipeline noticed.

Header labels can be wrong. One column headed “Type” holds what’s actually an acquisition method, while the real type sits elsewhere under a different name. Don’t carry a misleading source name into the bundle. Pick distinct keys and leave a comment explaining why they’re different.

Read the whole sheet. One of ours had the named concepts in the obvious rows and then, further up, a block of generic per-type guidance that’s used as the fallback when nothing more specific applies. Easy to miss entirely.

Files give you no constraints, so we put them in CI. Two checks, both blocking. Source coverage asserts that every meaningful value in the workbook shows up in the matching file, and reports it as a hard count rather than a pass/fail, because “the converter ran without error” tells you nothing about whether it captured anything. Spec conformance checks parseable frontmatter, the type key, the index rules, and that every relative link resolves.

Four consumers, one directory #

flowchart TB
  B["Served bundle<br/>(synced from git)"] --> A["REST API<br/>frontmatter to JSON"]
  B --> G[Agent]
  A --> M["MCP tools (read-only)"]
  A --> U["Governance UI + visualiser"]

Files are the bottom layer, and anything with filesystem access just reads them. The REST API is a thin thing that parses frontmatter and hands back JSON: bundle status, the root index, concept summaries, one concept in full, each rule document. It owns no data of its own, so it can’t disagree with the bundle. The MCP tools wrap those endpoints, so an IDE or a desktop client gets the same data with descriptions attached.

The MCP tool people actually use most is the boring one that reports path, concept count, last modified and format version. “Is what you’re reading current” turns out to be the first question in every debugging session.

Nothing writes to the served copy. Edits leave as pull requests and come back through sync, which is the next section.

Loading progressively instead of dumping the bundle #

The obvious thing to do with an agent is put the whole bundle in context. It works. It also makes every question pay for every concept, and it gives the model nothing to tell it which file is authoritative for the question in front of it.

The root index.md does that job instead. It lists every concept with its path, title and a one-line description, and the agent reads it first, picks a listed path, then opens that file plus whatever rule files the question actually needs.

Three rules keep it honest, and two of them exist because of specific failures:

It may only read a path that’s listed in the index. No slugifying the user’s phrase into a filename. A guessed path either 404s, which is fine, or opens a different concept that happens to exist, which is how you get a confidently wrong answer with a real file behind it.

Two plausible matches means asking, not picking. The one-line descriptions in the index are there mostly to make that distinction cheap.

Every fact gets tagged with where it came from: read from a file, inferred from the user’s wording, or missing. The mapping from an informal name to a concept can be inferred while the facts underneath stay governed, and that difference has to survive into the answer.

Token cost is the smaller win. The better one is that the files behind any given answer are a short list you can look at. Being able to answer “which sources produced this” is worth having before anyone formally asks you for it.

Versions and lineage #

Git is the version store. There isn’t a second one.

The service doesn’t clone the repo. It pulls a tarball of just the bundle path at a given ref, unpacks it into a temp directory, and swaps it into place atomically, recording the commit SHA and exposing it on the status endpoint. So any consumer can say exactly which version it’s reading. Stage the new tree next to the old one rather than writing over it, or a failed download leaves you serving half a bundle.

Changes come in one way:

flowchart LR
  E[Edit in governance UI] --> PR["Branch + PR"]
  PR --> RV[Review]
  RV -->|merged| W[Webhook]
  W --> SY["Pull tarball at new SHA<br/>atomic swap"]
  SY --> SV[Served bundle]

The webhook fires on merge to the base branch and triggers the sync. Nothing writes directly to the served copy, so what’s being served is always a merged commit rather than someone’s working state.

Lineage works in both directions and neither of them took any extra code. Downward, a value traces from a workbook cell through the converter to a file to the API field or the passage an agent read, and the coverage count is what makes that claim safe. Upward, every file has a git history, every change has a PR with a reviewer on it, and the copy being served is stamped with a SHA. History is git log. Releases are tags, which is how a consumer pins itself against in-flight edits. Blame works per line, on prose.

What it costs #

There’s no referential integrity. Two files can contradict each other, links rot, and CI only catches what I thought to check for. There’s no query layer, so “every concept where X and Y” means reading the whole bundle, which is fine at a few hundred files and obviously not at a few hundred thousand.

Anything generated from the bundle goes stale the moment the bundle changes. Our visualiser is a static snapshot baked at generation time, and more than once I’ve stared at a stale graph wondering why an edit didn’t show up before remembering to regenerate it and hard-refresh. Regenerate in the same step as conversion.

Concurrent edits are git’s model, conflicts and all, which is fine for prose and irritating for generated tables. And sync is a moving part now. A served copy that’s silently three days behind is worse than one that’s down, so the SHA and the last-sync time are both exposed and failures are loud.

I’d still make the same call. But it’s a call that depends on the corpus being small, closed and slow to change, and if ours stops being those things this post ages badly.