Skip to content

smartpipe

Semantic pipes and queries for your terminal.

Run PDFs, images, audio, video, and text through Unix verbs that understand their input: map, filter, top_k, reduce, and more. Log in with ChatGPT, bring a cloud API key, or use Ollama for local models. Either way it stays ordinary stdin/stdout composition.

cat reviews.txt \
| smartpipe filter "the reviewer is sarcastic" \
| smartpipe map "Extract {product, complaint, anger number: 0 to 1}"

Each line of reviews.txt is one item; --as file treats a whole file as one item (feeding smartpipe has the full table).

Pick your door

  • New to smartpipe? The Learn track - six short chapters from first pipeline to production habits (after installing).
  • Have a task in mind? The Cookbook - complete, copy-pasteable recipes: contracts, video, logs, training data.
  • Wondering how something works? Concepts - the item, ingestion, models, structured output.
  • Need the exact flag? The CLI reference - the whole surface, one page, contract-stable.

Try it on real files

No corpus handy? smartpipe-playground ships 26 MB of CC0 / public-domain practice files - invoices, reports, photos, recordings, screen sessions, and JSONL data - and smartpipe demo fetches it (checksum-verified, no model needed):

smartpipe demo
cd smartpipe-playground

smartpipe map "Extract {vendor, invoice_number, total number}" 'invoices/*.pdf'
smartpipe filter "the customer sounds frustrated" 'recordings/*.mp3'

Without smartpipe on the machine yet, the corpus is one line of curl away:

curl -L https://github.com/prabal-rje/smartpipe-playground/releases/download/v1/smartpipe-playground-v1.tar.gz | tar xz

Verbs

A verb is one operation on your data. Each reads stdin (or named FILES) and writes stdout, so verbs pipe into each other and into ordinary Unix tools.

Semantic verbs call a model:

Verb What it does Feels like
map transform each item with a prompt sed, but it understands
filter keep items matching a condition grep, but semantic
extend add extracted fields, keep the rest your record, plus columns
embed / top_k vectors; rank by similarity sort \| head, by meaning
reduce synthesize many items into one awk END, but literate
join match two inputs semantically SQL join, but semantic
cluster group by meaning, label each group themes with sizes and quotes
distinct fold near-duplicates; --exact is free sort -u, by meaning
diff what distinguishes two sets the post-incident answer
outliers the items least like the rest novelty, surfaced
graph corpus → entity/relationship graph; --fast is free by default the case wall, with citations

Free verbs never call a model. Run them first to cut the corpus before any paid stage:

Verb What it does Feels like
where filter on exact field predicates SQL WHERE
summarize count, average, percentiles, time buckets SQL GROUP BY
sort order items by a field sort
sample take a seeded random subset; --by stratifies shuf
agree inter-rater agreement: kappa, alpha, confusion the reviewers' number
getschema list fields, types, and coverage head, for structure
split break items into pieces (pages, minutes) split
chart terminal bars, SVG/PNG, facets, time series quick plots

Some semantic verbs have a conditionally free mode: join --on (key equality, no prompt), distinct --exact (hash-only folding), graph --fast (local NER, on-device), map/extend --dry-run (compose without sending), and smartpipe schema with a braces/DSL expression. Each stays at zero model calls by construction.

Concepts

Recipes & reference

Background

smartpipe brings the semantic-operator vocabulary of data frameworks like DocETL to Unix pipes, and adds file parsing, recursive chunking, and terminal-adaptive output. See the comparison for the landscape.