Build with Beam · docs that lie, caught by running them
DocDrift
DocDrift gives Beam an entire Python library (docs + code, up to ~400k tokens) in one call. Beam checks every suspicious doc example against the whole codebase, then writes a script that runs it. Across 12 popular libraries (bugs in 11 of 12 libraries; pydantic-settings had none) it found 34 real docs bugs; 30 crash exactly as the docs are written.
Beam read 12 whole Python libraries and ran their docs: 34 real bugs, 30 reproduce.
Voice: Kokoro-82M (Apache-2.0), generated locally
DocDrift gives Beam a whole Python library in one call, to find docs that lie.
Starlette's docs say: call receive JSON with mode binary. Run it as written, and it crashes with a TypeError.
One Beam call, the whole library: 394,577 prompt tokens. 3 candidates; 2 kept as real drift, 1 rejected.
A fix is filed upstream: pull request 3746, open, not merged.
Every card shows what the docs say, what the code says, and what running it gives: 34 real docs bugs in 11 of 12 libraries.
Does the 1M-token context help? Accuracy is about the same. But grounding is better: the right cross-file line 25 of 30 times, versus 19 at matched cost. Off-file evidence: 11 of 13, versus 6. p = 0.070: suggestive, not proven.
On the benchmark: 34 of 45 known drifts found; 30 crash exactly as the docs are written. 32 of the 34 bugs ship a script.
Then a fresh run on tomlkit, a library DocDrift had never seen.
2 findings: 1 verified version drift, and 1 arguable missing-imports NameError, in 96 seconds.
Run it on your repo with docdrift run, using a free Ollama key.
Recording — not a live call
Docs say one thing, the code does another. Worked example from starlette: docs call receive_json(data, mode="binary"); the implementation is receive_json(self, mode: str = "text"). Following the docs can raise TypeError.
Docs say / code says / running it gives. Permalinks point at the pinned commit.
Extra columns are the false/unclear/unlabelled split, among-labelled precision, extra-of-45, new AI-verified, and marginal token cost.
Each row is a script that ran the documented usage. Strict means TypeError, AttributeError, ImportError, or NameError on a documented-usage line. The starlette headline script was written by hand from Beam's verdict; the others were written by Beam.