Evaluate Suites

Evaluate Suites Module.

This module provides a helper function for evaluating different data partitions (suites) with different evaluator sets under a shared run_id and experiment tracker.

EvalSuite

Bases: BaseModel

Evaluation suite defining a data partition and its evaluators.

Attributes:

Name	Type	Description
data	str | BaseDataset | list[LLMTestCase] | list[ConversationalTestCase]	Input data as a string path, BaseDataset, or list of evaluation inputs (single-turn LLMTestCase or multi-turn ConversationalTestCase).
evaluators	list[BaseEvaluator]	List of evaluators to apply to this suite's data.
name	str | None	Optional name for the suite. If not provided, auto-generated as suite_0, suite_1, etc.

from_yaml(source, extra_dataset_kwargs=None) staticmethod

Build an EvalSuite from a single YAML suite document.

See gllm_evals/utils/yaml_suite_loader.py for the full YAML format, evaluator/metric resolution rules, ${VAR} substitution, and dataset resolution behavior.

Parameters:

Name	Type	Description	Default
source	str | Path	Path to the YAML suite document.	required
extra_dataset_kwargs	dict[str, Any] | None	Extra kwargs merged into dataset resolution, for values that can't be expressed in YAML. Defaults to None.	None

Returns:

Name	Type	Description
EvalSuite	EvalSuite	The constructed suite.

Raises:

Type	Description
ValueError	If the document is malformed, or an evaluator/metric/dataset reference is invalid.

from_yaml_dir(directory, pattern='*.y*ml', extra_dataset_kwargs=None) staticmethod

Build one EvalSuite per matching YAML file in a directory.

Parameters:

Name	Type	Description	Default
directory	str | Path	Directory containing one YAML file per suite.	required
pattern	str	Glob pattern for matching files. Defaults to ".yml".	'*.y*ml'
extra_dataset_kwargs	dict[str, Any] | None	Forwarded identically to every file. Defaults to None.	None

Returns:

Type	Description
list[EvalSuite]	list[EvalSuite]: One suite per matching file, sorted by path relative to directory.

Raises:

Type	Description
ValueError	If no files match, or any single file fails to load.

build_suite(**fields)

Builds an EvalSuite on the caller's behalf and normalizes a validation failure.

Constructing EvalSuite(...) directly keeps Pydantic's native ValidationError. An SDK operation that builds the suite from the caller's rows or configuration uses this helper instead, so the caller gets a normalized error with a code and without the submitted values.

Parameters:

Name	Type	Description	Default
**fields	Any	The EvalSuite fields.	{}

Returns:

Name	Type	Description
EvalSuite	EvalSuite	The validated suite.

Raises:

Type	Description
BaseError	The normalized error a validator raised, such as EvalInputError for mixed row types; otherwise the validation failure resolved for the input stage.

evaluate_suites(suites, experiment_tracker=None, batch_size=10, run_aggregators=None, dataset_name=None, run_id=None, raise_on_error=False, skip_on_missing_fields=False, **kwargs) async

Evaluate multiple suites with different evaluators under a shared run_id.

Allows different data partitions (suites) to use different evaluator sets while sharing one tracker and one run_id. Dataset names are namespaced per suite. Warns on duplicate test case content.

Parameters:

Name	Type	Description	Default
suites	list[EvalSuite]	List of evaluation suites, each with data and evaluators.	required
experiment_tracker	type[BaseExperimentTracker] | BaseExperimentTracker | None	Tracker class, instance, or None for CSV default.	None
batch_size	int	Batch size for evaluation (runner-level chunking). Defaults to 10.	10
run_aggregators	list[RunAggregatorCallable] | None	Custom run aggregators. Defaults to None.	None
dataset_name	str | None	Base dataset name. If None, auto-generated with timestamp.	None
run_id	str | None	Shared run ID. If None, auto-generated.	None
raise_on_error	bool	Stop the run at the first failed test case and raise its error. The error's debug_info holds suite and test_case_index, and batches the tracker saved before stay saved. Set to False to continue the run: each failed metric stays in the results with no score and is listed in failures. Defaults to False.	False
skip_on_missing_fields	bool	Skip metrics with incomplete input and record each skip in the result. Direct metric calls always reject incomplete input. Defaults to False.	False
**kwargs	Any	Additional configuration parameters. project_name (str), if passed here, is only used when experiment_tracker is a class or None -- an already-built tracker instance's own project_name always takes precedence (see _resolve_project_name_and_tracker()).	{}

Returns:

Name	Type	Description
SuiteExperimentResult	SuiteExperimentResult	Top-level result with per-suite results, pooled aggregators, the failed metrics in failures, and the scores that a fallback judge gave in fallbacks.

Raises:

Type	Description
EvalConfigurationError	If the resolved run_id contains a path separator. Raised before any suite is evaluated.
BaseError	The error of the first failed test case, if raise_on_error is True.
TypeError	If allow_batch_evaluation is passed.
EvalInputError	If a dataset is empty, or a test case has the wrong type or lacks required fields without explicit skip.
BaseError	Configuration, authentication, unknown-model, dataset, tracker, or result-contract errors. Service failures remain in metric results.