Skip to main content

Pipeline Methodology

This page summarises the scientific and technical choices behind each stage of the AstroSpectro pipeline, from raw data acquisition to model evaluation. Each stage links to the page that documents it in detail. The approach is iterative and data-driven.

1. Data download and management​

The pipeline starts with a reproducible acquisition step whose aim is a clean, locally tracked dataset.

  • Smart downloading: SmartDownloader retrieves FITS files in bulk with a round-robin strategy, so that the sample stays diverse across observation plans.
  • State management: log files (downloaded_plans.csv, trained_spectra.csv) record what was already downloaded or used for training, so no operation is repeated across sessions.
  • Local organisation: data lives in a predictable folder structure (data/raw/, data/catalog/).

See Data Download and LAMOST DR5.

2. Preprocessing and quality control​

Raw spectra are not directly usable.

  • Median normalisation: the flux is normalised by its median, a reference that is robust to outliers, so spectra become comparable.
  • Targeted smoothing: fine measurements such as FWHM use a Savitzky–Golay filter on line windows, reducing noise while preserving the line shape.
  • Missing values: features that cannot be measured stay missing in the feature table; the classifier imputes them (median by default) inside its pipeline.

See Preprocessing and Preprocessing guide.

3. Feature extraction​

The core of the approach is to extract physically interpretable information from each spectrum.

  • Line identification: the major lines (Hα, Hβ, Ca II K and H, Mg b, Na D) are located by the peak detector.
  • Physical measurements: each line is described by its strength (prominence, equivalent width) and width (FWHM).
  • Line ratios and indices: ratios of line strengths and band indices give temperature-sensitive indicators.
  • Observation metadata are not inputs. Position, redshift, SNR bands and instrument metadata can encode LAMOST's observing programmes rather than stellar physics, so the canonical spectro_only=True mode leaves them out (Physical Purity). That mode still does not remove every Gaia column: the reference run retains a few, as documented in the Data Leakage Audit.

See Feature Engineering and Feature Extraction guide.

4. Model training and validation​

  • Multi-model: SpectralClassifier supports several classifier families, so models can be compared under the same protocol.
  • Cross-validation and tuning: hyperparameters are searched with stratified cross-validation (GridSearchCV, or Optuna for larger searches).
  • Class imbalance: handled by balanced sample weights by default; resampling such as SMOTE is an optional setting, not part of the default configuration.

See Model Training and Classification.

5. Evaluation and reporting​

A model is only useful if its predictions can be understood and trusted.

  • Session reports: each training run writes a JSON report with metrics, model parameters and the files used, for traceability.
  • Standard figures: confusion matrices and calibration, ROC and precision–recall curves are generated per run; SHAP analyses document what the model relies on.

See Classification Performance and SHAP Interpretability.