Late-Fusion Multimodal ⭐
GalaxyFusionClassifier combines three inputs in separate branches that are concatenated late, just before a classification head. The question it tests: does explicit morphometry and photometry add information beyond what DINOv2 already encodes?
Architecture
| Branch | Input | Layers | Embedding |
|---|---|---|---|
| A — DINOv2 | 768 standardised features | 768 → 512 → 256 (MLP) | 256 |
| B — morphometry | C, A, S, Gini, M20 | 5 → 64 → 32 | 32 |
| C — photometry | g−r, u−r, Mr | 3 → 32 → 16 | 16 |
| Head | concatenation (304) | 256 → 128 → 10, BatchNorm, GELU, dropout | logits |
The full model has 643,034 parameters. Branches B and C are optional, and missing modalities are masked (embedding multiplied by a validity flag) instead of imputed: morphometry is available for 4,999 galaxies and the photometric triple for 5,651 (see Data Preparation). Training: 50 epochs, learning rate 3 × 10⁻⁴, dropout 0.35, label smoothing 0.1, AdamW with a cosine schedule.
Why late fusion: the modalities have very different dimensionality and coverage, and a late design lets each branch be switched off — which the ablation below does.
Results
_facts/astrovision/FACTS_INDEX.md §3 and shown when hovering over a number.Ablation: what do the extra modalities add?
| Configuration (30 epochs, test) | Balanced acc. | Δ vs DINOv2 only |
|---|---|---|
| DINOv2 only | 85.95% | +0.000 |
| DINOv2 + morphometry | 86.12% | +0.002 |
| DINOv2 + photometry | 85.90% | −0.001 |
| DINOv2 + morphometry + photometry | 85.75% | −0.002 |
The four configurations lie within about half a point of each other, and the full model (85.75%) is slightly below the DINOv2-only head (85.95%). The 50-epoch run of the same full architecture scored 86.31%: the spread between two trainings of one configuration is as large as the spread between configurations. There is no evidence that morphometry or SDSS colours improve classification.
This is consistent with the physical picture: concentration and colour are strongly correlated with morphology, so a representation that already encodes morphology has little left to gain from them, and both auxiliary modalities are missing for most galaxies.
Where the ~2 points over fine-tuned DINOv2 come from
Fusion scores roughly two points above fine-tuned DINOv2 (86.3% against 83.9%), yet the DINOv2-only branch already reaches 86.0%. The gain therefore comes from the MLP head trained on cached, standardised features — not from the added modalities. It is also not a like-for-like comparison:
- the fusion split is stratified with 2,661 test images; the DINOv2 evaluation used a random split with 2,660 — different image subsets;
- the cached DINOv2 features (
dino_features.npy) are, according to the notebook, extracted from the fine-tuned backbone. That backbone was trained on a different split, so some of the fusion test images may have been seen during fine-tuning. The data pack does not contain the feature file, so this cannot be verified here; until it is, the fusion score should be read as an upper bound.
Per-class behaviour
The run makes 332 errors on 2,661 test galaxies, 144 of them with a maximum probability above 0.8 — confident mistakes that motivate the uncertainty analysis. Disturbed has the lowest recall (0.59, precision 0.63).
The "Conclusions" table at the end of notebook 08 still reads "to be measured" for the ablation rows, although the ablation was run and saved afterwards.