Vision Bench · Autotagging experiment · 26 October 2026

Image autotagging: quality, latency and vocabulary coverage

What changes when you increase CAFormer's input resolution or use PixAI for the same tagging task? Five saved runs make the trade-offs visible on one fixed image selection.

1,024jointly successful images
5 pipelinessame frozen image selection
9,039common general tag outputs

I built Vision Bench for a recurring evaluation task: new models keep appearing, and I also fine-tune checkpoints. I wanted a small local tool that preserved complete predictions and let me revisit comparisons without rerunning the models. Image autotagging is one concrete use of that workflow.

Reading these results: the source tags are incomplete silver annotations. mAP and F1 measure agreement with those annotations; they are not percentages of correctly tagged images. Timing is one pass per configured pipeline.

What the saved runs show

Higher ranking score, higher measured cost

PixAI reached general mAP 0.5824 versus 0.5358 for the native CAFormer control. Its observed median latency was 223.6 ms versus 32.5 ms in these configurations.

Resolution did not improve results consistently

CAFormer direct 512 had the highest CAFormer general mAP, 0.5383. The native 384 control retained slightly higher general F1 and lower measured latency.

A larger vocabulary changes coverage

PixAI covered more source character-tag positives. Native vocabulary coverage and quality on common outputs answer different questions.

Quality and observed pipeline cost

General mAP and F1 below use the same 1,024 successful images and the same 9,039 common general outputs. F1 uses each profile's fixed threshold. No thresholds were selected on this set.

PipelineGeneral mAPGeneral F1Median msp95 msGPU GiB¹
CAFormer native 3840.53580.599832.541.70.975
CAFormer direct 3840.53130.595429.538.50.975
CAFormer direct 5120.53830.596743.652.21.014
CAFormer direct 6400.52370.580160.270.21.089
PixAI 1.0 Mexes0.58240.5997223.6247.96.270

¹ Sampled WDDM dedicated process-memory high water, in GiB, including loading and inference. Median and p95 use 1,019 successful warm rows per pipeline; the first five selected rows are excluded. CPU graph partitions are allowed by the ONNX CUDA profile.

Scatter plot: general mAP against measured median latency. PixAI has the highest measured mAP and latency; CAFormer direct 640 has lower mAP and higher latency than native 384.
Observed trade-offs between complete runtime pipelines. One pass, different runtime and input recipes, no confidence intervals. Download SVG.

Why native vocabulary matters

The common-output comparison contains 3,190 character outputs. The table below uses each model's complete native character vocabulary on the same successful images instead. Its denominator is all 1,687 source character-tag occurrences.

PipelineNative character outputsSource-label recallCoverage ceilingCovered source positives
CAFormer native 38432470.36100.4126696 / 1687
PixAI 1.0 Mexes83080.53410.62771059 / 1687

A vocabulary can only recall source positives it represents. The coverage ceiling counts represented source-positive occurrences before considering prediction scores. Broader coverage helps explain PixAI's higher native character recall; it does not establish that every additional prediction is correct.

Follow-up: character coverage with Danbooru aliases

On 19 November 2026, we reapplied the comparison to the same five saved runs and 1,024 images on CPU. We aligned model output names and source annotations using 13,791 active character aliases from the Danbooru API.

MatchingShared character tagsCovered source occurrencesCoverage
Exact names3,190696 / 1,68741.26%
Active aliases3,232703 / 1,68741.67%

Aliases recovered seven annotated occurrences on seven images. For example, they linked PixAI's aponia to CAFormer's aponia_(honkai_impact). This follow-up measures which source labels both models can represent before considering scores. It uses saved outputs and alias rules only. Use alias-aware comparison.

Experiment design

PipelineRuntimePinned model revisionFixed thresholds
CAFormer native 384PyTorch 2.11.0+cu128aac0699c88553d50eb673b41a81d8222936b22b2general 0.39; character 0.47
CAFormer direct 384PyTorch 2.11.0+cu128aac0699c88553d50eb673b41a81d8222936b22b2general 0.39; character 0.47
CAFormer direct 512PyTorch 2.11.0+cu128aac0699c88553d50eb673b41a81d8222936b22b2general 0.39; character 0.47
CAFormer direct 640PyTorch 2.11.0+cu128aac0699c88553d50eb673b41a81d8222936b22b2general 0.39; character 0.47
PixAI 1.0 MexesONNX Runtime CUDA 1.23.20800778563144a0e6fdf41ddadd84aae3cb0dbcfgeneral 0.17; character 0.27

CAFormer weights SHA-256: 75d656d3f3eec41753cd5553e088ea5d041203bf0e2696d6b4282a998ee43992. The aggregate JSON contains all model artifact hashes, thresholds, dataset fingerprints and recorded software/hardware metadata.

What this experiment supports

In these runs, PixAI produced the highest shared-vocabulary ranking score and broader native character coverage, at substantially higher observed latency and sampled process memory. CAFormer offered a lower-cost pipeline. Their general F1 values were numerically close at the configured thresholds, which does not establish equivalence.

Higher-resolution CAFormer configurations did not show a consistent benefit in this comparison: direct 640 had higher observed latency and sampled process memory while scoring below the original control on the reported general-tag metrics. These observations concern these weights, image selection and runtime recipes.

Reproduction and limits

For your own frozen dataset and saved runs:

vision-bench recompute ./runs/model-a
vision-bench recompute ./runs/model-b
vision-bench compare ./runs/model-a ./runs/model-b --output ./comparison.json
vision-bench export-report ./runs/model-a ./runs/model-b --title "Autotagging comparison" --output ./public-report
Full aggregate reportAggregate JSONCase study as MarkdownChecksums