Running STREAMLINE

STREAMLINE can be run through:

  • Google Colab

  • Local Jupyter notebook

  • Config-driven full pipeline runs

  • Phase-by-phase command-line calls

For most command-line work, start with the .cfg runner. It keeps shared settings, phase toggles, and phase-specific parameters in one editable file.

Choose A Run Path

Use case

Recommended path

Conference tutorial or first demo

Google Colab notebook

Interactive local exploration

STREAMLINE_Notebook.ipynb

Reproducible full pipeline run

python run.py -c run_configs/<config>.cfg

Debugging one phase

Phase CLI command

Faster local execution without Dask

run_cluster = Parallel

Local Dask execution

run_cluster = Local

HPC execution

BashSLURM, BashLSF, or a site-specific Dask cluster setting

Google Colab

Open the Colab notebook:

Open the STREAMLINE Colab notebook

At the top of the notebook, set the demo/run parameters. The notebook supports binary, multiclass, regression, and custom data modes.

Local Jupyter

From the repository root:

conda activate streamline
jupyter notebook

Open STREAMLINE_Notebook.ipynb. The top parameter block controls which demo or custom dataset is run.

Config-Driven Runs

Dry-run a config first:

python run.py -c run_configs/uci_binary_hcc.cfg --dry_run

Run a full binary demo:

python run.py -c run_configs/uci_binary_hcc.cfg

Run the multiclass and regression demos:

python run.py -c run_configs/uci_multiclass_student.cfg
python run.py -c run_configs/uci_regression_auto_mpg.cfg

The included configs are designed as reproducible examples. For a short notebook or Colab demonstration, use the notebook parameter block, which uses a smaller modeling budget and shows plots by default.

Partial-run examples:

python run.py -c run_configs/uci_binary_hcc.cfg --start_at p4
python run.py -c run_configs/uci_binary_hcc.cfg --stop_after p8
python run.py -c run_configs/uci_binary_hcc.cfg --only p6,p8,p11
python run.py -c run_configs/uci_binary_hcc.cfg --skip p3,p4

Config File Layout

Each config uses sections like:

[run]
output_path = out
experiment_name = UCIHCCPipeline
outcome_label = Class
outcome_type = Binary
instance_label = InstanceID
n_splits = 3
run_cluster = Serial
random_state = 42

[phases]
phase_order = p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11
do_p1 = True
do_p2 = True
do_p3 = True
do_p4 = True
do_p5 = True
do_p6 = True
do_p7 = True
do_p8 = True
do_p9 = True
do_p10 = True
do_p11 = True

[p6]
outcome_type = Binary
models = NB,LR,DT
scoring_metric = balanced_accuracy
metric_direction = maximize
n_trials = 200
timeout = 900

Use run_cluster = Local for a local Dask cluster, or run_cluster = Parallel for local joblib parallelism without Dask.

Parallel and Dask-backed runs show progress when tqdm/Dask progress support is available. Set STREAMLINE_PROGRESS=0 to disable these progress displays.

Use the included configs as templates:

  • run_configs/uci_binary_hcc.cfg

  • run_configs/uci_multiclass_student.cfg

  • run_configs/uci_regression_auto_mpg.cfg

Rerunning Phases

STREAMLINE stores resolved phase arguments in run_commands.pickle inside the experiment folder. If you rerun a phase and omit an option, the saved value can be reused. Explicit command-line or config values override the saved value.

Use this control when you need a clean rerun:

python -m streamline.p6_modeling.p6_cli \
  --output_path out \
  --experiment_name DemoBinary \
  --outcome_type Binary \
  --ignore_saved_run_command

Use --no_update_saved_run_command when you want to run with temporary settings without changing the saved command summary.

Phase CLI Commands

Each phase can also be run independently. A binary classification example:

python -m streamline.p1_data_process.p1_cli \
  --data_path data/UCIBinaryClassification \
  --output_path out \
  --experiment_name DemoBinary \
  --outcome_label Class \
  --outcome_type Binary \
  --instance_label InstanceID \
  --categorical_features data/UCIFeatureTypes/hcc_survival_categorical_features.csv \
  --quantitative_features data/UCIFeatureTypes/hcc_survival_quantitative_features.csv \
  --n_splits 3 \
  --force true

python -m streamline.p2_impute_scale.p2_cli --output_path out --experiment_name DemoBinary
python -m streamline.p3_feature_learning.p3_cli --output_path out --experiment_name DemoBinary
python -m streamline.p4_feature_importance.p4_cli --output_path out --experiment_name DemoBinary
python -m streamline.p5_feature_selection.p5_cli --output_path out --experiment_name DemoBinary

python -m streamline.p6_modeling.p6_cli \
  --output_path out \
  --experiment_name DemoBinary \
  --outcome_label Class \
  --outcome_type Binary \
  --instance_label InstanceID \
  --models NB,LR,DT \
  --scoring_metric balanced_accuracy \
  --metric_direction maximize

python -m streamline.p7_ensembles.p7_cli --output_path out --experiment_name DemoBinary
python -m streamline.p8_summary_statistics.p8_cli --output_path out --experiment_name DemoBinary
python -m streamline.p9_compare_datasets.p9_cli --output_path out --experiment_name DemoBinary

python -m streamline.p10_replication.p10_cli \
  --rep_data_path data/UCIRepBinaryClassification \
  --dataset_for_rep data/UCIBinaryClassification/hcc_survival.csv \
  --output_path out \
  --experiment_name DemoBinary

python -m streamline.p11_reporting.p11_cli \
  --experiment_path out/DemoBinary \
  --report_mode standard

python -m streamline.p11_reporting.p11_cli \
  --experiment_path out/DemoBinary \
  --report_mode replication

More examples are available in sample_runcommands.txt.

Discovery Commands

Several phases can list registry options:

python -m streamline.p2_impute_scale.p2_cli --output_path out --experiment_name DemoBinary --list-imputers
python -m streamline.p2_impute_scale.p2_cli --output_path out --experiment_name DemoBinary --list-scalers
python -m streamline.p3_feature_learning.p3_cli --output_path out --experiment_name DemoBinary --list-learners
python -m streamline.p4_feature_importance.p4_cli --output_path out --experiment_name DemoBinary --list-models
python -m streamline.p6_modeling.p6_cli --output_path out --experiment_name DemoBinary --outcome_type Binary --list_models
python -m streamline.p7_ensembles.p7_cli --output_path out --experiment_name DemoBinary --list_ensembles