Skip to content

outliers - the items least like the rest

top_k's mirror: instead of "nearest to my query", farthest from everything. Novelty, surfaced - the failure shape you haven't seen, the mislabeled training row, the alert that isn't like the others.

cat today.log \
| smartpipe outliers 5
# → outliers: median neighbor distance 0.21 - these are 3.1x-3.9x out
# → {"text": "kernel: watchdog: BUG: soft lockup CPU#3", "__distance": 0.81, "source": "line 48122"}

Embeddings only - no chat calls. The score is each item's mean cosine distance to its nearest neighbors (robust when the corpus has several normal clusters); the stderr line anchors it against the corpus median so the distance has context.

Output records mirror top_k's shape (__distance where top_k has __score; original fields survive for JSON rows, {"text": …} for plain lines).

Small print: needs at least 3 items; rows that fail to embed are excluded with a warning (items that fail to embed can't be scored).

Typical loops: smartpipe where 'level has "error"' | smartpipe outliers 5 (triage novel failures), smartpipe outliers 20 < train.jsonl (hunt label noise before a training run).

With an ocr-model configured, ingested PDFs and images parse through it at ingestion (one item per page, disclosed per row; --ocr-model overrides per run) - so a folder of scans can join the corpus as text. Unset, nothing changes.