Vision Bench · Frozen selection · Protocol v1

Repeat the 1,024-image autotagging experiment

The exact sample IDs, image hashes, annotation snapshot, five model profiles and saved predictions behind the PixAI 1.0 / CAFormer comparison.

Both archives are versioned and hash-checked. Image bytes and model weights are downloaded separately from their sources.

1. Replay the metrics on a CPU

Extract the reproduction kit into a new directory. From that directory, use an activated Python 3.13 virtual environment:

python -m pip install -r requirements-replay.txt
python reproduce.py verify
python reproduce.py fetch-reference
python reproduce.py replay

Expected: "native_metrics_exact": true and "paired_comparison_exact": true for all five saved runs. This path needs no images, weights, GPU or account.

Reference score shards, sample order, tag vocabularies and prediction records are retained byte for byte. Published run manifests omit local paths, launcher tokens and process IDs. Replaying historical latency records does not measure your machine's performance.

2. Recover the exact inputs

python reproduce.py fetch-images --audit image-availability.json
python -m vision_bench validate --dataset dataset.json
python reproduce.py fetch-models

The downloader uses frozen Danbooru URLs and rejects changed or missing bytes. The snapshot includes 887 sample JPEGs and 137 original files; using larger originals instead of the sample variants changes the experiment.

CAFormer is gated upstream: obtain access on its model page and authenticate with your own HF account before downloading. PixAI is public. Exact model commits and artifact hashes are recorded in reproduction.json.

Image bytes total 147 MB. See the publication-time image availability audit. Upstream availability can change; no replacement images or fresh annotations are silently accepted.

3. Rerun the five CUDA pipelines

Use separate virtual environments for the two runtimes. The full instructions include the pinned installations, access requirements and exact commands.

In the CAFormer environment:

python reproduce.py run --pipeline caformer --measure-memory

In the PixAI environment:

python reproduce.py run --pipeline pixai --measure-memory
python reproduce.py compare-new

Kit 1.0.1 pins cuDNN 9.10.2.21 for PixAI; the newer 9.27 failed in the checked FP16 pipeline. Omit --measure-memory on Linux. All 1,024 images must succeed; the common evaluation vocabulary must match. Fresh inference reports quality deltas rather than assuming a cross-hardware tolerance. GPU timings and byte-identical fresh predictions are not promised.

What stays fixed

Dataset fingerprint:
6a0e70f23bb08d427f284dbddef21d84b26e63e1b32b3321d939fa174eadd79b

This is an existing balanced holdout with incomplete source labels. It does not estimate natural prevalence, and training overlap has not been established. Adding another model can change the common vocabulary; preserve this protocol when comparing against these results.

Publication checks recovered all 1,024 exact image files and completed five fresh inference runs. General/character mAP and micro F1 deltas were all zero on the checked machine. Read the validation record.

Kit checksums · Provenance · Browse all files on HF