Evaluate
Evaluate Module.
.. deprecated::
evaluate is deprecated in favor of :func:gllm_evals.evaluate_suites.evaluate_suites.
It is kept only as a thin backward-compatible shim: every call is routed straight through
evaluate_suites with a single-suite wrapper, and the result is adapted back to
:class:~gllm_evals.types.ExperimentResult. New code should build an
:class:~gllm_evals.evaluate_suites.EvalSuite and call evaluate_suites directly.
The evaluation pipeline requires input data to already contain model outputs
(e.g., actual_output). It does not perform live model inference.
evaluate(data, evaluators, experiment_tracker=None, batch_size=10, allow_batch_evaluation=False, run_aggregators=None, **kwargs)
async
Evaluate precomputed model outputs.
.. deprecated::
Use :func:gllm_evals.evaluate_suites.evaluate_suites instead. This function now
routes directly through evaluate_suites by wrapping data and evaluators
in a single :class:~gllm_evals.evaluate_suites.EvalSuite, then adapts the
SuiteExperimentResult back to an ExperimentResult. A logger warning is
emitted on every call.
Input data must already contain model outputs (e.g. actual_output).
This function does not perform live model inference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
str | BaseDataset | list[LLMTestCase]
|
The data to evaluate. |
required |
evaluators
|
list[BaseEvaluator]
|
The evaluators to use. |
required |
experiment_tracker
|
BaseExperimentTracker | None
|
The experiment tracker to use. |
None
|
batch_size
|
int
|
The batch size to use for evaluation (runner-level chunking for memory management). Defaults to 10. |
10
|
allow_batch_evaluation
|
bool
|
Enable batch processing mode for LLM API calls. When True, the runner passes entire chunks to evaluators for batch processing. Defaults to False. |
False
|
run_aggregators
|
list[RunAggregatorCallable] | None
|
Custom run aggregators to compute batch-level statistics. Each callable receives (evaluation_results, data) and returns a dict of summary metrics. Defaults to None. |
None
|
**kwargs
|
Any
|
Additional configuration parameters forwarded to |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
ExperimentResult |
ExperimentResult
|
Structured result containing evaluation results and experiment URLs/paths. |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |