Model Benchmarking

Overall comparison

Model	Accuracy	Balanced acc.
Random baseline (10 balanced classes)	—	10.0%
Majority class	14.9%	10.0%
SimpleCNN (scratch)	67.2%	59.5%
DINOv2 linear probe (frozen)	63.6%	55.5%
EfficientNet-B0 fine-tuned	84.5%	82.7%
DINOv2 ViT-B/14 fine-tuned	86.5%	83.9%
Fusion head (single run, other test split)	87.5%	86.3%

Test sets: first four image models on the seed-42 random split (2,660 images); fusion on a stratified split (2,661 images).

Provenance. Model scores in this section are copied from the output cells of the AstroVision notebooks (and, for DINOv2 per-class recall, read from an exported figure). The documentation data pack contains no metrics file for them, so they cannot be re-derived from the pack; the cell of origin is listed for each value in _facts/astrovision/FACTS_INDEX.md §3 and shown when hovering over a number.

Balanced accuracy

SimpleCNN59.5%

DINOv2 probe55.5%

EfficientNet-B082.7%

DINOv2 fine-tuned83.9%

Fusion (single run)86.3%

The progression is not monotonic with model size: the frozen 86 M-parameter DINOv2 (55.5%) is below a 0.65 M-parameter CNN (59.5%), and fine-tuning is what makes DINOv2 competitive. The margin of fine-tuned DINOv2 over EfficientNet-B0 is 83.9% against 82.7% — about one point, on about 2,660 test images and one training run each; it supports "comparable or slightly better", not a ranking.

Per class

Class	SimpleCNN	EfficientNet-B0	DINOv2 FT	Fusion
0 · Disturbed	0.11	0.44	0.53	0.59
1 · Merging	0.55	0.89	0.86	0.90
2 · Round Smooth	0.90	0.90	0.94	0.95
3 · In-between Round Smooth	0.85	0.95	0.95	0.91
4 · Cigar Shaped Smooth	0.12	0.83	0.72	0.86
5 · Barred Spiral	0.68	0.84	0.87	0.91
6 · Unbarred Tight Spiral	0.63	0.78	0.83	0.83
7 · Unbarred Loose Spiral	0.61	0.76	0.78	0.79
8 · Edge-on without Bulge	0.75	0.96	0.96	0.93
9 · Edge-on with Bulge	0.76	0.92	0.93	0.97

Recall on the test set. DINOv2 values are read from the exported confusion matrix (2 decimals).

Across models, the ordering of classes by difficulty is broadly stable: Disturbed is the hardest, followed by Cigar Shaped Smooth (for the CNNs), Unbarred Loose Spiral and Unbarred Tight Spiral; edge-on and smooth classes are easiest. Notably, adding modalities in the fusion model does not rescue Disturbed (recall 0.59).

How comparable are these numbers?

Issue	Consequence
Two different test splits (2,660 random versus 2,661 stratified images)	Fusion versus the other models is an inter-split comparison
One training run per model; no seed repetition	Differences of 1–2 points are within plausible run-to-run variation (the fusion configuration scored 86.3% in one run and 85.7% in another)
Selection by validation balanced accuracy, reporting on test	Fine; but the validation curve of DINOv2 is noisy
Fusion features possibly from a backbone fine-tuned on part of its test set	Fusion score possibly optimistic (see Late-Fusion)