Pramen

One static binary. One YAML file. Columnar data movement with governed LLM enrichment — from object storage into PostgreSQL, crash-safe and budgeted.

Quickstart View on GitHub

3.1×faster than psql \copy — native binary COPY, measured on 5M rows

~3M rows/sSQL filter + derivation throughput under a hard memory ceiling

100%AI result reuse on replay — completed inference is never re-billed

1static binary — no services, drivers, or toolchains to install

Why Pramen

Columnar all the way through

Apache Arrow record batches move through sources, SQL transforms, and sinks without row-by-row conversion. Backpressure is structural: memory in flight is bounded no matter how large the input is.

Governed AI, not vibes

Semantic transforms are schema-bound and validated. Every inference is recorded in a durable, content-addressed ledger — a crash or re-run reuses completed results instead of paying for them twice.

Spend control before dispatch

Hard token budgets per record, enforced before a model is called. Provider batch APIs are scheduled for you when latency allows, at batch pricing.

Honest delivery semantics

At-least-once by default, loads inside a single transaction: a failed run leaves your table untouched. The delivery contract is documented, tested, and enforced by the runtime.

A pipeline is one file

apiVersion: pramen.dev/v1alpha1
kind: Pipeline
metadata:
  name: enrich-tickets
spec:
  source:
    type: object_store
    url: /data/tickets/
    format: { type: parquet }
  transforms:
    - id: normalize
      type: sql
      query: SELECT ticket_id, lower(trim(description)) AS description FROM input
  sink:
    type: postgres
    target: support.tickets

pramen validate pipeline.yaml   # every problem reported at once, with paths
pramen explain pipeline.yaml    # the resolved plan
pramen run pipeline.yaml        # bounded, transactional, Ctrl-C safe