19 open-source projects,
every number checkable

Unit timed: one csvstat run over a 550k-row CSV (end to end)

$ python csvkit/utilities/csvstat.py big.csv

Bit-identical: every reported statistic equals stock csvkit

Unit timed: one random graph through six analyses (betweenness, PageRank, clustering, …)
A library with no entry point: the clock is on a benchmark script over its public API.

$ python ao_bench.py

Every reported field identical across 10 checked graphs

Unit timed: one week of a climate summary chain (coarsen → rolling → groupby_bins → anomaly → daily max → NetCDF write)
A library with no entry point: the clock is on a benchmark script over its public API.

$ python ao_bench.py

Statistics within 1.1e-5 absolute of the stock chain

Unit timed: one test image: JPEG decode → transform → generator → PNG written to disk

$ python test.py --dataroot datasets/horse2zebra/testA --name horse2zebra_pretrained --model test --no_dropout

Worst pixel differs by 2 uint8 levels (PSNR 64.9 dB), less than the stock program differs from itself

Unit timed: one sample (200 generated tokens, pythia-410m, bf16)

$ python litgpt/__main__.py generate checkpoints/EleutherAI/pythia-410m --max_new_tokens 200 --num_samples 8

Bit-identical against the frozen bf16 reference, pinned and held-out prompts

Unit timed: one batch of series fitted and forecast by StatsForecast
A library with no entry point: the clock is on a benchmark script over its public API.

$ python ao_bench.py

Bit-identical forecasts

Unit timed: one sentence synthesised end to end (phonemiser → Tacotron2-DDC → vocoder → waveform)

$ python TTS/bin/synthesize.py --text "The quick brown fox jumps over the lazy dog, again and again." --model_name tts_models/en/ljspeech/tacotron2-DDC --out_path out.wav --use_cuda

Bit-identical waveform (PSNR 124 dB) on pinned and held-out sentences

Unit timed: one input image: decode → 4x RRDBNet (fp16) → post-process → encode + write

$ python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs

PSNR 59.8 dB against the stock output, under one 8-bit code

Unit timed: one encode() request (a list of texts → unit-norm embeddings), as the example script issues it

$ python examples/sentence_transformer/applications/computing-embeddings/computing_embeddings.py

Cosine 0.99999 against the fp32 output

Unit timed: one output frame of the 10-minute-tutorial trailer rendered end to end (decode → effects → composite → ffmpeg encode)

$ python docs/_static/code/getting_started/moviepy_10_minutes/trailer.py

At most 1 code of pixel difference (PSNR 51.7 dB)

Unit timed: one image through detect.py: read → letterbox → yolov5s forward → NMS → annotated write

$ python detect.py --source data/images --weights yolov5s.pt

Bit-identical on pinned images and holdout

Unit timed: one streamed clip through the server path (fresh client → segments → retirement)
A library with no entry point: the clock is on a benchmark script over its public API.

$ python ao_bench.py

Bit-identical transcripts

Unit timed: one clip scored with detect-adaptive end to end (decode → per-frame score → cut decision → scene list)

$ python scenedetect/__main__.py -i demo.mp4 detect-adaptive list-scenes -n

Bit-identical scores and an identical cut list

Unit timed: one image + prompt through demo/inference_on_a_image.py (SwinT backbone, BERT, deformable decoder → boxes)

$ python demo/inference_on_a_image.py -c groundingdino/config/GroundingDINO_SwinT_OGC.py -p weights/groundingdino_swint_ogc.pth -i .asset/cat_dog.jpeg -o out -t "cat ear."

Boxes move by less than 0.1 px

Unit timed: one short clip interpolated 2x end to end: decode → SSIM check → IFNet → encode

$ python inference_video.py --video=demo.mp4 --exp=1

Worst pixel 1 code off (PSNR 57.7 dB)

Unit timed: one training iteration of train.py (shakespeare_char)

$ python train.py config/train_shakespeare_char.py --compile=False --max_iters=400 --eval_interval=200 --log_interval=10

Loss trajectory within 8e-5 relative, gradient cosine ≥ 0.99999

Unit timed: one LoRA fine-tuning step of VibeVoice-ASR on the toy dataset (decode → tokenise → collate → forward/backward), measured on an RTX 5090

$ python finetuning-asr/lora_finetune.py --model_path microsoft/VibeVoice-ASR --data_dir finetuning-asr/toy_dataset --output_dir out --num_train_epochs 3 --per_device_train_batch_size 1 --learning_rate 1e-4 --bf16 --report_to none

Loss within 6.5e-2 relative, gradient cosine 0.885; a switch restores the stock path, the user decides

Unit timed: one training step of train.py on a batch of 16 coco128 images

$ python train.py --data coco128.yaml --weights yolov5s.pt --img 640 --epochs 3 --batch-size 16

Loss within 2e-3 relative, gradient cosine ≥ 0.996

Unit timed: one audio file transcribed (--model small, float16, no alignment)

$ python whisperx/__main__.py audio.wav --model small --compute_type float16 --no_align --output_dir out

Features within 0.016 of the stock implementation

Unit timed: one series: TimeSeries → Scaler → NBEATSModel → fit (3 epochs) → forecast
A library with no entry point: the clock is on a benchmark script over its public API.

$ python ao_bench.py

Loss within 2e-3 relative, MAPE unchanged