Guide: Data Download

This guide shows how to use the pipeline's interactive interface to retrieve LAMOST DR5 spectra. The underlying logic is in the SpectraManager and SmartDownloader modules.

How it works

Downloading is orchestrated by the notebooks/01_download_spectra.ipynb notebook. Under the hood it uses the SmartDownloader class, which implements reproducible logic:

1. State management: it checks data/catalog/downloaded_plans.csv to skip observation plans that were already fully downloaded in previous sessions.
2. File scraping: for each new plan, it scrapes the official LAMOST web page to list all available .fits.gz files.
3. Round-robin downloading: to keep the dataset diverse, it alternates between plans (one spectrum from plan A, then one from plan B, and so on) instead of downloading them one after another.
4. Progress bar: an interactive progress bar tracks the download.
5. State update: once the session is complete, it updates the log with the plans that are now complete.

Practical use, step by step

The process is driven by an interactive interface.

1. Open the management notebook

Make sure your virtual environment (venv) is activated, then open notebooks/01_download_spectra.ipynb.

2. Configure and launch

Run the first code cell. An interactive interface appears:

• Plans: the number of new observation plans to process.
• Spectra: the maximum total number of spectra to download during this session.

Then click Start download.

3. Monitor and verify

Track progress with the progress bar. Once finished, check that:

• new subfolders and .fits.gz files have appeared in data/raw/,
• data/catalog/downloaded_plans.csv has been updated if some plans were fully completed,
• a detailed session log file has been created in logs/.

Safe cleanup option

The notebook also provides an interactive cell to clean the data/raw/ directory. To avoid accidental deletion, this operation requires textual confirmation and first creates a full backup in data/raw_backup/.