Open source · MIT · Python 3.11+

Vision Bench

Image tagger benchmarks & VLM experiments

Compare new image autotaggers and fine-tuned checkpoints locally. Save complete predictions, recompute metrics and compare runs offline.

Runs on your computer. Supply model weights and evaluation images separately.

Published comparison: PixAI 1.0 vs CAFormer

Five configured pipelines from two model families on the same 1,024 Danbooru-derived images, with zero failed samples. The table summarizes general-tag quality and measured pipeline costs.

General tags · 9,039 shared labels · 1,024 paired images · measured 26 October 2026
Pipeline / runtimemAP ↑Micro F1 ↑Median
ms/image ↓
Sampled GPU
GiB
CAFormer B36 · native 384PyTorch · AMP FP160.53580.599832.490.98
CAFormer B36 · direct 384PyTorch · AMP FP160.53130.595429.530.98
CAFormer B36 · direct 512PyTorch · AMP FP160.53830.596743.621.01
CAFormer B36 · direct 640PyTorch · AMP FP160.52370.580160.231.09
PixAI 1.0 · 1008ONNX Runtime · FP160.58240.5997223.636.27

mAP averages shared labels with positive support. Micro F1 uses the common representable vocabulary at fixed thresholds: CAFormer 0.39, PixAI 0.17. All CAFormer rows use unchanged FP32 weights with AMP FP16; native 384 is the original two-stage preprocessing control, while direct variants use a single-stage resize.

Windows 11 / RTX 3080 Ti. PyTorch 2.11.0 + CUDA 12.8 for CAFormer; ONNX Runtime 1.23.2 with CUDA for PixAI. Median latency covers decoding, preprocessing and inference over 1,019 rows after the first five, excluding model loading and integrity checks. GPU memory is sampled dedicated process high water including loading. One timing pass per pipeline; background GPU applications were running.

Source tags are incomplete silver annotations. Unlabeled predictions are not verified false positives, and training overlap has not been checked. Frozen sample IDs, annotations, exact image recovery and saved predictions are available in the reproduction kit. These observations apply to this selection and these runtime configurations.

Detailed autotagging report with quality, latency and character vocabulary coverage

Supported tagger models

Bundled, hash-pinned ONNX profiles cover the models below. Profile support and published results are separate: the comparison above evaluates CAFormer B36 and PixAI 1.0.

Model artifacts and images are supplied locally under their own terms.
Model sourceBundled supportPublic comparison
CAFormer B36FP32 and AMP FP16 ONNX profilesIn this comparison
PixAI 0.9FP32 ONNX profileProfile available
PixAI 1.0 (Mexes export)FP16 ONNX profileIn this comparison
WD EVA02 large v3FP32 ONNX profileProfile available
CL tagger v2ONNX profileProfile available
Camie tagger v2FP32 ONNX profileProfile available

Optional PyTorch CAFormer profiles support controlled resolution experiments, including the variants above. Custom profiles accept your own local artifacts and fine-tuned checkpoints. Read the tagging guide.

VLM experiments

Run versioned image-chat tasks against your own HTTP server. Save responses, validate strict JSON schemas, measure repeat agreement and export samples for human review.

There is no published VLM quality comparison here yet. Response validity and agreement measure format and stability; task accuracy requires reference labels or human review. Read the VLM guide.

Analyze saved runs in 0.4.0

Compare per-image scores and paired intervals, fit thresholds on a calibration split, and export difficult images for review.

On the published runs, alias matching raised shared character coverage from 41.26% to 41.67%. Read the CPU follow-up.

Download the 0.4.0 wheel and source

Try the CPU workflow

Clone the source and create a Python 3.11+ environment:

git clone https://github.com/TimeOfAnarchy/vision-bench.git
cd vision-bench
python -m venv .venv

Activate it with .\.venv\Scripts\Activate.ps1 in Windows PowerShell or source .venv/bin/activate on Linux/macOS, then install:

python -m pip install ".[cpu,demo]"

The offline CPU example generates images and a tiny untrained ONNX model to exercise the complete workflow. Its toy scores do not measure real model quality. This static Space presents the tool and reports; inference runs locally.