Vision Bench
Image tagger benchmarks & VLM experiments
Compare new image autotaggers and fine-tuned checkpoints locally. Save complete predictions, recompute metrics and compare runs offline.
Runs on your computer. Supply model weights and evaluation images separately.
Published comparison: PixAI 1.0 vs CAFormer
Five configured pipelines from two model families on the same 1,024 Danbooru-derived images, with zero failed samples. The table summarizes general-tag quality and measured pipeline costs.
| Pipeline / runtime | mAP ↑ | Micro F1 ↑ | Median ms/image ↓ | Sampled GPU GiB |
|---|---|---|---|---|
| CAFormer B36 · native 384PyTorch · AMP FP16 | 0.5358 | 0.5998 | 32.49 | 0.98 |
| CAFormer B36 · direct 384PyTorch · AMP FP16 | 0.5313 | 0.5954 | 29.53 | 0.98 |
| CAFormer B36 · direct 512PyTorch · AMP FP16 | 0.5383 | 0.5967 | 43.62 | 1.01 |
| CAFormer B36 · direct 640PyTorch · AMP FP16 | 0.5237 | 0.5801 | 60.23 | 1.09 |
| PixAI 1.0 · 1008ONNX Runtime · FP16 | 0.5824 | 0.5997 | 223.63 | 6.27 |
mAP averages shared labels with positive support. Micro F1 uses the common representable vocabulary at fixed thresholds: CAFormer 0.39, PixAI 0.17. All CAFormer rows use unchanged FP32 weights with AMP FP16; native 384 is the original two-stage preprocessing control, while direct variants use a single-stage resize.
Windows 11 / RTX 3080 Ti. PyTorch 2.11.0 + CUDA 12.8 for CAFormer; ONNX Runtime 1.23.2 with CUDA for PixAI. Median latency covers decoding, preprocessing and inference over 1,019 rows after the first five, excluding model loading and integrity checks. GPU memory is sampled dedicated process high water including loading. One timing pass per pipeline; background GPU applications were running.
Source tags are incomplete silver annotations. Unlabeled predictions are not verified false positives, and training overlap has not been checked. Frozen sample IDs, annotations, exact image recovery and saved predictions are available in the reproduction kit. These observations apply to this selection and these runtime configurations.
Supported tagger models
Bundled, hash-pinned ONNX profiles cover the models below. Profile support and published results are separate: the comparison above evaluates CAFormer B36 and PixAI 1.0.
| Model source | Bundled support | Public comparison |
|---|---|---|
| CAFormer B36 | FP32 and AMP FP16 ONNX profiles | In this comparison |
| PixAI 0.9 | FP32 ONNX profile | Profile available |
| PixAI 1.0 (Mexes export) | FP16 ONNX profile | In this comparison |
| WD EVA02 large v3 | FP32 ONNX profile | Profile available |
| CL tagger v2 | ONNX profile | Profile available |
| Camie tagger v2 | FP32 ONNX profile | Profile available |
Optional PyTorch CAFormer profiles support controlled resolution experiments, including the variants above. Custom profiles accept your own local artifacts and fine-tuned checkpoints. Read the tagging guide.
VLM experiments
Run versioned image-chat tasks against your own HTTP server. Save responses, validate strict JSON schemas, measure repeat agreement and export samples for human review.
There is no published VLM quality comparison here yet. Response validity and agreement measure format and stability; task accuracy requires reference labels or human review. Read the VLM guide.
Analyze saved runs in 0.4.0
Compare per-image scores and paired intervals, fit thresholds on a calibration split, and export difficult images for review.
On the published runs, alias matching raised shared character coverage from 41.26% to 41.67%. Read the CPU follow-up.
Download the 0.4.0 wheel and sourceTry the CPU workflow
Clone the source and create a Python 3.11+ environment:
git clone https://github.com/TimeOfAnarchy/vision-bench.git
cd vision-bench
python -m venv .venv
Activate it with .\.venv\Scripts\Activate.ps1 in Windows PowerShell or source .venv/bin/activate on Linux/macOS, then install:
python -m pip install ".[cpu,demo]"
The offline CPU example generates images and a tiny untrained ONNX model to exercise the complete workflow. Its toy scores do not measure real model quality. This static Space presents the tool and reports; inference runs locally.