join - match two inputs, semantically

Merge stdin against a second input wherever a plain-English predicate holds. The SQL join's semantic cousin: no keys, no exact equality - meaning.

cat tickets.jsonl \
| smartpipe join "ticket {left.text} concerns product {right.name}" --right products.jsonl
# → {"left": {"text": "the laser printer keeps smoking"}, "right": {"name": "LaserJet 9"}, "__score": 0.91, "__sources": [{"path": "-", "as": "lines", "line": 3}, {"path": "products.jsonl", "as": "lines", "line": 12}]}

How it works (and why it's affordable)

A naive semantic join would ask the model about every pair - 1,000 × 1,000 = a million calls. smartpipe does embed → block → judge:

1. Embed the --right file once (it's read whole and indexed in memory).
2. Block: each stdin item is embedded and matched to its --k nearest right-side candidates by similarity - pure math, no model calls.
3. Judge: only those candidate pairs go to the chat model, with a yes/no verdict prompt.

Cost is lines × k, never lines × right-size. Before the first judge call, smartpipe tells you the worst case on stderr (when it exceeds a couple hundred):

join: 1,204 left items · up to 5 candidates each = at most 6,020 model calls (cap with --max-calls)

The predicate

Braces name a side's field - every brace must pick a side:

• {left.text} / {right.text} - the whole item (for JSON items without a text field, the raw line).
• {left.body}, {right.name}, … - a JSON field; a pair whose item lacks the field is skipped with a warning naming what it does have.
• A predicate must mention both sides - one that reads only one side would match everything or nothing, so it's refused up front.

Options

Option	Meaning
--right FILE	The finite side to index (JSONL or plain lines; with an ocr-model set, a PDF/image parses to page items). Required; never stdin
--k N	Candidates judged per left item (default 5 - the recall knob, see below)
--threshold FLOAT	Similarity floor (0-1) a candidate must clear before judging
--model TEXT	Chat model for the judge calls
--embed-model TEXT	Embedding model for both sides
--fields A,B	Project output columns - field paths reach the sides: --fields left.id,right.name,__score
--on 'left.K == right.K'	Key-equality join (repeatable, AND-ed). Alone = free deterministic join; with a predicate = blocking. K is a field path into each side's record (left.order.sku)
--output FORMAT	auto · json · csv · tsv
--concurrency N / --max-calls N	Simultaneous outbound API calls / hard billable-unit ceiling
--ocr-model TEXT	Parse ingested PDFs/images with a document parsing model - both sides, --right included (the role)

Output

One record per matched pair, in left-input order, a left item's matches consecutive and ranked by similarity:

{"left": {…}, "right": {…}, "__score": 0.87, "__sources": [{…left's ref…}, {…right's ref…}]}

__sources (item 64) is the pair's provenance: a two-element list with each parent's spine ref (left's, then right's) in the compact __source form - --on-only joins carry it too. --bare strips it with the rest of the __ spine.

The sides stay nested (never flat-merged) so identical field names on both sides can't corrupt each other. Exit codes are the usual contract: 0 all judged (zero matches is still success, like grep), 1 some pairs or lines skipped, 2/64 per the taxonomy.

--k is a recall knob, not a performance knob

Candidates that blocking drops are matches the judge never sees. On a synthetic 40×40 corpus with known ground truth (make join-eval):

k	recall of true matches
1	0.20
3	0.56
5 (default)	0.85
10	1.00

Real numbers depend on your embedding model and data. The spot-check: rerun a sample with --k 20 --threshold 0 and compare match counts - a jump means the default is dropping true matches; raise --k (and consider a stronger embedding model).

Items bigger than the window

Oversized sides (left or right) are no longer skipped: their chunks are embedded once, mean-pooled for blocking, and the judge reads only the most-relevant chunk of the oversized side (highest similarity against the other side), row-disclosed as oversized → best-chunk judge. A 300-page spec in the right file matches tickets without any judge call ever seeing 300 pages.

The unmatched remainder

--unmatched FILE writes every left item that matched nothing, verbatim, one line each - your worklist for a looser second pass (bigger --k, softer predicate, or a human). A final stderr note reports the split: join: 34 matched · 7 unmatched → leftovers.txt.

Streaming

The left side streams flag-free, like every per-item verb - so join is a live enrichment operator:

tail -f events.log \
| smartpipe join "event {left.text} involves customer {right.name}" --right customers.jsonl

The right side can never stream (an index can't be built from a tail): --right - is a usage error that says so.

See also

• Cookbook: live stream enrichment
• top_k - ranking against one query instead of matching two sets
• .sem files - save a join as an executable stage

Join kinds

--kind picks which set the pipe receives (default inner, today's matched-pairs behavior):

cat orders.jsonl \
| smartpipe join "the same purchase" --right invoices.jsonl --kind anti
# → {"id": 4411, "customer": "acme", "total": 240.0}    ← unmatched LEFT rows, verbatim
… --kind leftouter \
| head -1
# → {"left": {...}, "right": null}                       ← every left row, match or not

anti emits the unmatched left rows directly - "orders with no invoice" - as passthrough so they pipe onward (into cluster, a CSV, a ticket). leftouter keeps every left row with "right": null where nothing matched. The summary line works for every kind.

Writing predicates a judge can satisfy

The judge is strict at temperature 0: a predicate that demands certainty ("is the same purchase") gets false on any paraphrase. Phrase the claim as the evidence supports it:

• Weak: "{left.desc} is the same purchase as {right.item}"
• Strong: "order {left.desc} and invoice {right.item} name the same product"
• Strong: "{left.text} plausibly describes a defect in {right.name}"

Words like names, describes, concerns, and plausibly judge the relationship in the text; is the same judges an identity the text usually can't prove. If a join matches nothing, try the predicate by hand on one pair with map first.