`top_k` - rank by similarity¶

Sorts items by semantic closeness to a query and returns the best matches. Like sort | head, but by meaning instead of by string.

Examples¶

# Find the 5 most relevant résumés:
cat resumes/*.txt \
| smartpipe top_k 5 --near "distributed systems engineer"

# Rank a precomputed corpus (embed once, query many times):
cat corpus.embeddings \
| smartpipe top_k 10 --near "Q3 revenue strategy"

# Threshold mode: everything above a similarity of 0.8, no fixed count:
cat articles.jsonl \
| smartpipe top_k --near "climate policy" --threshold 0.8

# The three-stage pipeline: embed, rank, extract:
cat legal/*.txt \
| smartpipe embed \
| smartpipe top_k 20 --near "indemnification" \
| smartpipe map "Extract {clause_text, liability_cap number}"

How it works¶

top_k embeds your query and every item, then ranks the items by cosine similarity. Give it a number (K), a --threshold, or both:

top_k 5 --near Q - the 5 closest items.
top_k --near Q --threshold 0.8 - everything scoring ≥ 0.8.
top_k 5 --near Q --threshold 0.8 - up to 5 items that also score ≥ 0.8.

Each result gains a __score from 0 to 1 (higher is closer): a JSON field for JSON items, or a trailing tab-separated column for plain text.

Reusing embeddings¶

If an item already carries a vector (because it came from smartpipe embed), top_k uses it directly instead of re-embedding - so you can embed a large corpus once and run many queries against it cheaply:

cat docs/*.md \
| smartpipe embed > corpus.embeddings
cat corpus.embeddings \
| smartpipe top_k 5 --near "first question"
cat corpus.embeddings \
| smartpipe top_k 5 --near "second question"

The vector field is an internal field, so top_k drops it from the output and keeps the rest of the record plus __score.

Embed rows also carry an __embedder stamp - the model that produced the vector. top_k refuses a corpus whose stamp differs from the model this run resolves to (same dimensions or not, two models' vectors live in different spaces), naming both models and the fix. Rows from older releases without the stamp keep working, with one calm note.

Streaming: `--stream` (the live leaderboard)¶

Keep a running top-K over a live stream:

tail -f tickets.jsonl \
| smartpipe top_k 5 --stream --near "billing dispute"

At a terminal the K-line board repaints in place as better matches arrive. In a pipe, every membership/order change emits a JSONL snapshot: a {"__snapshot": N} marker line, then the K records in rank order, each with __score and __rank - split on the markers to consume programmatically; no change means no output. --stream needs K, reads stdin only, and skips (rather than fails on) a record whose embedding dimensions don't match the query.

One honest edge: --stream with a media-embed-model that stands apart from embed-model. The query has to embed before the stream's composition is knowable, so it embeds in the text space - and the first image item that then arrives would land in the joint space, so the run stops with the one-run-one-vector-space error rather than mix two geometries. Batch mode doesn't have this problem: it sees the whole corpus before embedding the query, so a media-bearing corpus ranks your text query in the joint space automatically. For a live board over media, set embed-model to the joint model so both roles agree - or keep the stream text-only.

Options¶

Option	Meaning
`K` (positional)	Return at most this many items
`--near TEXT`	The query to rank against (required)
`--threshold FLOAT`	Keep everything at or above this similarity (0-1)
`--stream`	Live leaderboard over a stream (needs `K`)
`--embed-model TEXT`	The embedding model
`--media-embed-model TEXT`	A JOINT text+image embedder for media items - a text query then ranks an image corpus in its space (the role)
`--ocr-model TEXT`	Parse ingested PDFs/images with a document parsing model (the role)
`--concurrency N`	Max parallel model calls
`--fields A,B`	Select + order columns of JSON results (incl. `__score`) (details)

(Spellings top-k and topk also work.)

Performance¶

Items that need embedding are sent in chunks of up to 64 texts per call (precomputed vector fields from smartpipe embed records are reused, never re-embedded). A failed chunk is retried item by item, so one bad item skips alone. --stream stays one item per call - streaming processes one item at a time to keep latency low.

Gotchas¶

Use one embedding model for a corpus and its queries. If the corpus was embedded with a different model than the query, top_k stops with an error naming both models (the __embedder stamp catches even same-dimension mismatches; older unstamped corpora fall back to the dimension check).
top_k buffers everything. Unlike map/filter, it must see every item to rank, so it isn't a streaming operation.

top_k - rank by similarity¶