Governed AI enrichment

Semantic transforms treat a language model call the way a database treats a write: as an operation with a contract — fixed inputs, a fixed instruction, a typed output schema, hard budgets, and a durable record.

The inference ledger

Every unit of semantic work gets a content-addressed work key: a hash over the canonicalized inputs, instruction, model identity, output schema, and prompt revision. Before anything is sent to a provider, the ledger is consulted; after a result validates, it is recorded durably.

The consequences fall out naturally:

Crash safety. Kill the process mid-run and restart: completed work is found in the ledger and skipped. Measured in the spike: zero lost results across kill points, 100% reuse on replay.
Incremental re-enrichment. Re-run over a grown dataset and only new or changed records incur model cost. Change the prompt revision and exactly the affected work re-executes.
Provenance. Every value can answer: which model, which prompt revision, which provider request, how many tokens, validated against which schema.

Ledger overhead was measured at microseconds per work item — noise next to any model call.

Reuse contract (RQ2)

The ledger’s behaviour is a fixed contract, not a best-effort cache. Formal statement and offline measurements live in docs/research/rq2-memoization.md:

Work key = hash of canonical inputs + instruction + schema + provider/model + params.
Completed recorded results are immutable — replays never re-dispatch or re-bill them.
Crash after provider-batch submit reconciles by job/item id (zero rebill).
Review-queued records stay withheld across replays with zero re-dispatch until accept/reject.
Measured (mock + SQLite): 100% reuse / 0 tokens on online replay; 90% savings on a 200-row / 20-unique duplicate workload; incremental re-enrichment bills only changed and new keys.

Regenerate figures with ./scripts/rq2-memoization.sh (or mise run rq2-memoization).

Budgets, ceilings, and the circuit breaker

Budgets are enforced before dispatch, not reported after. Per-record: maxInputTokensPerRecord gates each record on a conservative estimate; maxOutputTokensPerRecord becomes the provider-side generation cap.
maxRunTokens is a hard per-run ceiling on provider-reported tokens for the transform. Before each dispatch, the tokens already consumed plus the next request’s worst case are projected against it; crossing it fails the run with a clear message. Ledger reuse costs nothing against the ceiling, so a re-run of mostly-cached data proceeds even under a tight cap.
The circuit breaker is always armed: a run aborts after breaker.maxConsecutiveInvalid consecutive invalid outputs (default 25). A spike like that is almost always systemic — a broken prompt revision, wrong model, degraded endpoint — and under onInvalid: drop each invalid record still costs real tokens.
Output is validated against the declared schema — types, nullability — and lands as properly typed Arrow columns, not a JSON string you parse downstream. Invalid output follows an explicit policy: fail the run, drop and count, or route to review.

The review queue

onInvalid: review is a durable routing, not a euphemism for drop. The failed record — its inputs, the reason, and the raw model output — lands in a queue inside the same ledger database, and three guarantees hold:

Withheld, everywhere: a queued record never reaches the sink, in this run or any replay, until a human decides.
Never re-billed while waiting: replays see the queue entry and skip dispatch entirely — an undecided record costs zero tokens forever.
Decisions obey the same contract as models: an accepted correction (pramen ai review accept) is validated against the declared output schema, then recorded in the ledger as a completed result attributed to human-review with zero tokens — from then on the record flows through every run like any other cached result. A rejection is a permanent, auditable drop.

The workflow lives in the pramen ai review CLI: list, export (JSONL for labeling tools), accept, reject.

Online and batch execution

Both dispatch shapes run today, under identical governance:

Online (execution: online, or auto without dispatch hints): one provider call per ledger miss, streamed row by row.
Provider-batch (execution: batch, or auto when the cost model recommends it): ledger misses are collected while input streams through, submitted as one asynchronous job, then polled, fetched, validated, and joined back to the buffered rows. The job id is recorded per item in the durable ledger before results are awaited, so a run that crashes after submission reconciles on restart by job and item id instead of resubmitting — submitted work is never billed twice. Provider batch APIs (Bedrock, OpenAI, and compatible) typically price at ~50% of online rates.

execution: auto runs the online-vs-batch cost model when the transform sets dispatch.expectedRecords and dispatch.deadlineSeconds (optional token and rate-card overrides). It picks the cheaper mode that still meets the deadline. Without those hints, auto stays online. Plan or sweep with pramen ai dispatch-plan; the published mock/stub frontier is in Dispatch policy.

The batch machinery — submission, polling, reconciliation, budgets — is fully exercised against the local batch-capable mock provider today; the Bedrock and OpenAI batch adapters are the cloud legs of P1.8 and plug into the same operator.

Providers

The provider layer is deliberately neutral, and three adapters ship today: mock (deterministic offline output for dry-runs and tests), openai-compat (any OpenAI-protocol endpoint — including self-hosted vLLM or Ollama for residency-constrained deployments), and bedrock (Amazon Bedrock Converse, AWS default credential chain, region pinned per model declaration). The same budgets, validation, ledger, and provenance apply identically to all of them; an adapter only translates requests.