Higher ranking score, higher measured cost
PixAI reached general mAP 0.5824 versus 0.5358 for the native CAFormer control. Its observed median latency was 223.6 ms versus 32.5 ms in these configurations.
What changes when you increase CAFormer's input resolution or use PixAI for the same tagging task? Five saved runs make the trade-offs visible on one fixed image selection.
I built Vision Bench for a recurring evaluation task: new models keep appearing, and I also fine-tune checkpoints. I wanted a small local tool that preserved complete predictions and let me revisit comparisons without rerunning the models. Image autotagging is one concrete use of that workflow.
Reading these results: the source tags are incomplete silver annotations. mAP and F1 measure agreement with those annotations; they are not percentages of correctly tagged images. Timing is one pass per configured pipeline.
PixAI reached general mAP 0.5824 versus 0.5358 for the native CAFormer control. Its observed median latency was 223.6 ms versus 32.5 ms in these configurations.
CAFormer direct 512 had the highest CAFormer general mAP, 0.5383. The native 384 control retained slightly higher general F1 and lower measured latency.
PixAI covered more source character-tag positives. Native vocabulary coverage and quality on common outputs answer different questions.
General mAP and F1 below use the same 1,024 successful images and the same 9,039 common general outputs. F1 uses each profile's fixed threshold. No thresholds were selected on this set.
| Pipeline | General mAP | General F1 | Median ms | p95 ms | GPU GiB¹ |
|---|---|---|---|---|---|
| CAFormer native 384 | 0.5358 | 0.5998 | 32.5 | 41.7 | 0.975 |
| CAFormer direct 384 | 0.5313 | 0.5954 | 29.5 | 38.5 | 0.975 |
| CAFormer direct 512 | 0.5383 | 0.5967 | 43.6 | 52.2 | 1.014 |
| CAFormer direct 640 | 0.5237 | 0.5801 | 60.2 | 70.2 | 1.089 |
| PixAI 1.0 Mexes | 0.5824 | 0.5997 | 223.6 | 247.9 | 6.270 |
¹ Sampled WDDM dedicated process-memory high water, in GiB, including loading and inference. Median and p95 use 1,019 successful warm rows per pipeline; the first five selected rows are excluded. CPU graph partitions are allowed by the ONNX CUDA profile.

The common-output comparison contains 3,190 character outputs. The table below uses each model's complete native character vocabulary on the same successful images instead. Its denominator is all 1,687 source character-tag occurrences.
| Pipeline | Native character outputs | Source-label recall | Coverage ceiling | Covered source positives |
|---|---|---|---|---|
| CAFormer native 384 | 3247 | 0.3610 | 0.4126 | 696 / 1687 |
| PixAI 1.0 Mexes | 8308 | 0.5341 | 0.6277 | 1059 / 1687 |
A vocabulary can only recall source positives it represents. The coverage ceiling counts represented source-positive occurrences before considering prediction scores. Broader coverage helps explain PixAI's higher native character recall; it does not establish that every additional prediction is correct.
On 19 November 2026, we reapplied the comparison to the same five saved runs and 1,024 images on CPU. We aligned model output names and source annotations using 13,791 active character aliases from the Danbooru API.
| Matching | Shared character tags | Covered source occurrences | Coverage |
|---|---|---|---|
| Exact names | 3,190 | 696 / 1,687 | 41.26% |
| Active aliases | 3,232 | 703 / 1,687 | 41.67% |
Aliases recovered seven annotated occurrences on seven images. For example, they linked PixAI's aponia to CAFormer's aponia_(honkai_impact). This follow-up measures which source labels both models can represent before considering scores. It uses saved outputs and alias rules only. Use alias-aware comparison.
| Pipeline | Runtime | Pinned model revision | Fixed thresholds |
|---|---|---|---|
| CAFormer native 384 | PyTorch 2.11.0+cu128 | aac0699c88553d50eb673b41a81d8222936b22b2 | general 0.39; character 0.47 |
| CAFormer direct 384 | PyTorch 2.11.0+cu128 | aac0699c88553d50eb673b41a81d8222936b22b2 | general 0.39; character 0.47 |
| CAFormer direct 512 | PyTorch 2.11.0+cu128 | aac0699c88553d50eb673b41a81d8222936b22b2 | general 0.39; character 0.47 |
| CAFormer direct 640 | PyTorch 2.11.0+cu128 | aac0699c88553d50eb673b41a81d8222936b22b2 | general 0.39; character 0.47 |
| PixAI 1.0 Mexes | ONNX Runtime CUDA 1.23.2 | 0800778563144a0e6fdf41ddadd84aae3cb0dbcf | general 0.17; character 0.27 |
CAFormer weights SHA-256: 75d656d3f3eec41753cd5553e088ea5d041203bf0e2696d6b4282a998ee43992. The aggregate JSON contains all model artifact hashes, thresholds, dataset fingerprints and recorded software/hardware metadata.
In these runs, PixAI produced the highest shared-vocabulary ranking score and broader native character coverage, at substantially higher observed latency and sampled process memory. CAFormer offered a lower-cost pipeline. Their general F1 values were numerically close at the configured thresholds, which does not establish equivalence.
Higher-resolution CAFormer configurations did not show a consistent benefit in this comparison: direct 640 had higher observed latency and sampled process memory while scoring below the original control on the reported general-tag metrics. These observations concern these weights, image selection and runtime recipes.
For your own frozen dataset and saved runs:
vision-bench recompute ./runs/model-a
vision-bench recompute ./runs/model-b
vision-bench compare ./runs/model-a ./runs/model-b --output ./comparison.json
vision-bench export-report ./runs/model-a ./runs/model-b --title "Autotagging comparison" --output ./public-report