Reproducibility
This page documents what AstroSpectro records for every run so that a result on this site can be traced back to the exact model, data file and software versions that produced it — and, where the data are available, re-run. It covers both the classifier sessions and the PHY-3500 dimensionality-reduction notebooks.
Why this matters
Results on this site are quoted with a run identifier. That identifier is only useful if it points to files that fix the model, the input table, the configuration and the environment. Without them, a metric cannot be checked, and two runs cannot be compared.
Run identifiers
Every run is named by a UTC timestamp in the format YYYYMMDDTHHMMSSZ, generated by utc_now_tag() (src/utils.py) or make_timestamp() (src/dimred/run_reporter.py). UTC avoids ambiguities between machines in different time zones. The reference classifier, for example, is run 20260522T161236Z.
What a classifier session writes
MasterPipeline.run_training_session and SpectralClassifier.save_model produce:
| File | Content |
|---|---|
data/models/spectral_classifier_<model>_<ts>.pkl | The trained SpectralClassifier object |
data/models/spectral_classifier_<model>_<ts>_meta.json | Save time (UTC), Python / NumPy / scikit-learn / XGBoost versions, model type, prediction target, best_params_, class labels, candidate and selected feature names, training file (trained_on_file) |
data/reports/<ts>/session_report_<model>_<ts>.json | Model path and MD5 hash of the .pkl, number of spectra processed, train/test sizes, feature lists, parameters, accuracy, balanced accuracy, classification report, confusion matrix, ROC-AUC, average precision, Brier score, class filter, experiment name and notes |
data/reports/<ts>/*.png | Confusion matrix, ROC, precision–recall and calibration curves (feature importance when enabled) |
data/reports/last_config_used.json | The full configuration chosen in the training interface (overwritten by each session) |
For the reference run, the meta file records Python 3.11.9, NumPy 2.3.4, scikit-learn 1.7.2 and XGBoost 3.1.1, the training file features_20260522T153123Z.csv, and the model hash e752bb7ce5494364364f77a4200aa480.
The MD5 hash identifies the exact model file: recomputing it with md5sum() (src/utils.py) on a downloaded .pkl tells whether it is the model the metrics refer to.
from src.utils import md5sum
md5sum("data/models/spectral_classifier_xgboost_<ts>.pkl") # compare with model_hash_md5
Random seeds
| Component | Seed |
|---|---|
SpectralClassifier (split, CV folds, selector, sampler, threshold tuning) | random_state=42 by default, passed to every random component |
| Reference XGBoost model | random_state = 42 (best_model_params) |
EmbeddingEngine (UMAP, t-SNE) | random_state=42 by default |
| Embedding stability | stability_report(n_seeds=…) re-runs the embedding with several seeds and compares them by Procrustes distance |
UMAP and t-SNE are not deterministic across seeds; their stability is measured rather than assumed (see Dimensionality Reduction).
Data provenance
- Feature tables are written as
data/processed/features_<ts>.csv; the model meta file names the table used for training. - Spectra already used are recorded in
data/catalog/trained_spectra.csvbyDatasetBuilder, so later batches only draw new spectra. - Two data generations (G1 and G2) are kept apart on the site; see Data & Datasets.
Dimensionality-reduction runs
The PHY-3500 notebooks save their results through src/dimred/run_reporter.py (save_pca_run, save_umap_tsne_run, save_autoencoder_run):
| File | Content |
|---|---|
data/reports/runs/<family>/<family>_run_<ts>.json | Structured report: shapes, parameters, metrics, timings, figure inventory |
data/reports/runs/<family>/<family>_run_<ts>.txt | Human-readable summary of the same run |
<family>_run_<ts>.joblib | Fitted objects (embeddings, models) |
<family>_run_latest.{json,txt,joblib} | Copy of the most recent run, for the next notebook |
The latest files are convenience aliases and are overwritten by each run: always cite the timestamped file.
Experiment tracking (optional)
Weights & Biases logging is opt-in (use_wandb=True in MasterPipeline or the training interface). should_use_wandb() returns False if the flag is off, if WANDB_DISABLED=true, or if the wandb package is missing; init_wandb_run() opens a run in the AstroSpectro project. W&B calls never interrupt training. The files listed above are the reference record; W&B is a convenience layer on top of them. See Experiment Tracking.
How to reproduce a published run
- Find the run identifier on the page (run provenance card or figure caption).
- Open its
_meta.json: install the recorded package versions and retrieve the feature table named intrained_on_file. - Re-train with the same configuration (parameters in
best_model_params, class filter,spectro_only), or load the saved.pkland check its MD5 against the session report. - Compare the new session report with the published one (balanced accuracy, confusion matrix).
Limits: the raw LAMOST spectra and the feature tables are not distributed with the documentation, and last_config_used.json only keeps the last session, so the exact configuration of an older run must be read from its own session report.