Skip to content

2 · Structured data

Chapter 1 ended with braces: {vendor, total} turned prose into JSON. This chapter teaches the whole ladder - from a two-second shorthand to a schema a production pipeline can trust.

Braces name the fields you want

echo 'Invoice from Acme Corp, dated 2026-01-15, total $1250' \
| smartpipe map "Extract {vendor, date, total number}"
# → {"vendor": "Acme Corp", "date": "2026-01-15", "total": 1250}

The prompt is still ordinary prose: everything outside the braces is the instruction the model follows ("Extract", here); the braces declare the fields to return.

A bare field means "any scalar". Add a type when it matters, a ? when the answer may honestly be absent, and a plain-English description when the model needs a hint:

smartpipe map "Extract {vendor string: the legal name, due date: the invoice due date, total number}"

When the instruction outgrows one line, keep it in a file and pass @extract.md (or --prompt-file extract.md) - braces inside the file are still live grammar.

Types: string · number · integer · boolean · date · datetime · enum(a, b, …) · string[] · number[] · {a, b}[]. A date field always comes back as YYYY-MM-DD (a datetime as full ISO-8601) no matter how the model phrases it, so where 'due >= "2026-01-01"', sort --by due, and summarize 'count() by bin(due, 1d)' all work downstream. And enum is the workhorse for labels:

cat tickets.jsonl \
| smartpipe map "Classify this ticket. Add {label enum(bug, praise, request)}" --tally label

--tally label keeps a live count on the status line and prints the final distribution - the fastest sanity check that your labels make sense.

Lists of objects

When one item holds several records - events, entities, line items - an inner brace group followed by [] extracts them all at once, typed:

echo 'Kickoff Jan 15 2026 went fine; the Mar 3 2026 launch was rough' \
| smartpipe map "List {events {name string, when date, severity enum(low, high)}[]}"
# → {"events": [{"name": "kickoff", "when": "2026-01-15", "severity": "low"}, {"name": "launch", "when": "2026-03-03", "severity": "high"}]}

Inner fields take the same types and : guidance as outer ones, and inner date fields still canonicalize. Add --explode events to get one row per event, ready for where and summarize. One rule: object lists nest one level deep - if the inner objects need structure of their own, flatten it or extract in two passes.

Records in, records out

Feed JSONL and extend adds your extracted fields WITHOUT dropping anything:

head -3 tickets.jsonl \
| smartpipe extend "Add {sentiment enum(pos, neg, neutral)}"
# → {"id": 812, "body": "app crashes when saving", "sentiment": "neg"}

Even a plain-prompt map over records answers with records ({"result": …} plus provenance) - rows never silently flatten to prose.

When braces aren't enough

The schema ladder, cheapest rung first:

  1. Braces - free, instant, right there in the prompt.
  2. --schema-from DSL - free, adds constraints: --schema-from 'vendor string; total number >= 0'.
  3. smartpipe schema EXPR - compiles braces or the DSL into a JSON Schema file you can review, --check data.jsonl against, or --example. Free.
  4. --schema FILE - full JSON Schema, enforced with one repair retry.
smartpipe schema '{vendor string, total number}' > invoice.json
smartpipe map "Extract the details" --schema invoice.json 'invoices/*.pdf'

Rather build it interactively? Bare smartpipe schema at a terminal opens a small workshop - /add fields, /test a data file, /save the schema, all free (reference).

When a reply fails validation even after the repair, the row is skipped and counted - or kept as a machine-readable failure marker with --keep-invalid (chapter 5).

Shaping what leaves the pipe

  • --fields name,email picks and orders columns.
  • --output csv / tsv / json forces a format; the default adapts (blocks at a terminal, JSONL in a pipe).
  • --bare strips the __ metadata when you redirect with >.

Next: 3 · Files and media - PDFs, images, audio, and the granularity dial.