Skip to content

Cookbook

Real pipelines, copy-pasteable. Each recipe is a small composition of smartpipe verbs with the Unix tools you already use (or don't yet - the five-line toolbox intro covers them).

Recipe What it does
Contract & document extraction Pull structured fields out of a folder of PDFs
Invoice reconciliation Scanned invoices become rows; an anti-join finds what never hit the ledger
Training-data prep The curator's loop: dedupe, gate, label at scale, decontaminate
Video Q&A and scene digests Ask questions of video - and video RAG over a folder of recordings
Knowledge graph Who and what connects across a mixed corpus - free NER first, cited edges always
Meeting digests A week of call recordings into one digest that cites recording and minute
Customer feedback Detractor themes, week-over-week drift, deck-ready charts
.sem stage files Save a pipe stage as an executable script
Live stream enrichment Tag a live stream with the catalog rows it concerns (join)
Log triage Filter, cluster, and diff noisy logs by meaning
Ranking documents Find the most relevant files for a query
Live monitoring tail -f through semantic verbs, windows, and a live leaderboard
Research methods Stratified samples, inter-rater agreement, run manifests, and the --local-only fence

Try the recipes on real files

Every recipe assumes a folder of your own data, but none requires one: smartpipe-playground ships 26 MB of CC0 / public-domain invoices, reports, photos, recordings, screen sessions, and JSONL data. Just run:

smartpipe demo
cd smartpipe-playground

smartpipe map "Extract {vendor, invoice_number, total number}" 'invoices/*.pdf'
smartpipe embed 'sessions/*.mp4' > sessions.embeddings

(smartpipe demo is free - no model calls, checksum-verified. The same corpus is one curl -L https://github.com/prabal-rje/smartpipe-playground/releases/download/v1/smartpipe-playground-v1.tar.gz | tar xz if you'd rather fetch it by hand.)

The shape of every recipe

smartpipe verbs are filters - they read stdin (or files), write stdout, and compose:

cat data \
| smartpipe filter "..." \
| smartpipe map "Extract {...}" \
| jq ... > out.csv

Because structured output is JSONL, everything downstream of a map speaks jq, csv, spreadsheets, or another smartpipe verb. Because plain output is just text, everything upstream can be grep, head, find, or git.

A note on cost

Every verb calls a model once per item (plus a repair retry only when structured output needs fixing). If you're on a paid API, head-limit while you iterate:

cat big.jsonl \
| head -20 \
| smartpipe map "..."    # test on 20 before running 20,000

Or stay free with a local Ollama model - see Models & providers.