Setup and Configuration¶

The setup module provides factory helpers that translate configuration dicts into initialized model/evaluator instances and dataset specifications.

Dataset Configuration¶

DatasetSpec

class DatasetSpec(NamedTuple)¶

Immutable specification for one dataset in a multi-dataset run.

Variables:

name (str) – Unique dataset identifier (validated at config load time to match ^[A-Za-z0-9_-]+$). Used as the dataset column value in output files and report section headers.
prompts (list) – List of base prompts (strings or dicts) loaded from the dataset entry’s base_prompts or base_prompts_file. Guaranteed non-empty (zero-prompt datasets are rejected at preflight).
evaluator – Evaluator instance (BaseEvaluator | None) for this dataset. May be None if setup_evaluator failed during config initialization; the pipeline’s preflight check raises ValueError if any dataset has evaluator=None when Stage 3 is enabled.