Evaluator

Evaluator module for orchestrating evaluation workflows.

This module provides evaluator classes that coordinate metrics and evaluation logic for different use cases. Evaluators handle the evaluation process, including data preparation, metric execution, and result aggregation.

Available evaluators: - BaseEvaluator: Abstract base class for all evaluators - BaseGenerationEvaluator: Shared generation evaluator base class - GEvalGenerationEvaluator: G-Eval based generation evaluation - ClassicalRetrievalEvaluator: Traditional retrieval evaluation methods - LMBasedRetrievalEvaluator: LM-based retrieval evaluation methods - CompositeEvaluator: Combines multiple metrics into a single evaluation unit - QTEvaluator: Question-answering evaluation - AgentEvaluator: Combined agent trajectory and generation evaluation - SummarizationEvaluator: Summarization evaluation methods

`AgentEvaluator(models=None, metrics=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

Bases: BaseEvaluator

Evaluator for agent tool calling and generation quality evaluation with MetricsAggregator.

This evaluator combines: 1. DeepEval Tool Correctness metric for agent tool call evaluation 2. GEval generation metrics (completeness, groundedness, redundancy) 3. MetricsAggregator for polarity-aware binary scoring aggregation

Metrics are filtered at runtime via can_evaluate — any metric whose required_fields are absent from the input data is skipped with a warning.

Expected input (LLMTestCase with generation fields): - agent_trajectory (list[dict[str, Any]]): The agent trajectory - expected_agent_trajectory (list[dict[str, Any]]): Expected trajectory - input (str): The input query for generation evaluation - actual_output (str): The generated response - expected_output (str): The expected response - retrieved_context (str | list[str] | None, optional): The retrieved context

Aggregation Logic

Uses MetricsAggregator.aggregate() for base aggregation:
- aggregate_success = all(metric.success) — AND-gate across all metrics
- aggregate_score = mean(quality_score) — quality_score = score (higher_is_better=True) or 1 - score (higher_is_better=False)
Trajectory accuracy metric results are included in output but do not affect aggregation

Output Structure:

The output is a flat dictionary containing:

- Aggregated Results:
    - aggregate_explanation (str): Human-readable explanation of the evaluation
    - aggregate_success (bool): AND-gate of all metric success values
    - aggregate_score (float): Mean quality score with polarity inversion

- Tool Call Evaluation Results:
    - deepeval_tool_correctness (dict): Tool correctness evaluation result containing
    score, success, threshold, and other contract fields

- Generation Evaluation Results:
    - completeness, groundedness, redundancy: Individual metric results with full contract fields

- Trajectory Accuracy Results (optional, only when metric evaluates available data):
    - langchain_agent_trajectory_accuracy (dict): Trajectory accuracy evaluation result

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`metrics`	`list[BaseMetric]`	The configured metrics for evaluation.
`metrics_aggregator`	`MetricsAggregator`	Aggregator for polarity-aware binary scoring.

Example

import os from gllm_evals.evaluator import AgentEvaluator from gllm_evals.metrics.tool_use.deepeval_tool_correctness import DeepEvalToolCorrectnessMetric from gllm_evals.metrics.tool_use.langchain_agent_trajectory_accuracy import ( ... LangChainAgentTrajectoryAccuracyMetric ... )

Configure metrics — trajectory accuracy runs conditionally via can_evaluate

tool_correctness = DeepEvalToolCorrectnessMetric() trajectory_accuracy = LangChainAgentTrajectoryAccuracyMetric()

evaluator = AgentEvaluator( ... metrics=[tool_correctness, trajectory_accuracy], ... )

result = await evaluator.evaluate(data)

Initialize the AgentEvaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. When provided, propagated to every metric instance. Defaults to None.	`None`
`metrics`	`list[BaseMetric] \| None`	Pre-configured metric instances. If None, defaults to DEFAULT_METRICS. Metrics whose required_fields are absent from the input data are skipped at runtime via can_evaluate. Defaults to None.	`None`
`aggregation_method`	`AggregationSelector \| None`	The aggregation method to use for each metric. Defaults to None.	`None`
`max_concurrent_judges`	`int \| None`	The maximum number of concurrent judges per metric. Defaults to None.	`None`
`run_parallel`	`bool`	Whether to run metrics in parallel. Defaults to True.	`True`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120.	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every default metric. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If models list contains invalid invoker configurations.

`required_fields` `property`

Returns the intersection of required fields from all registered metrics.

Uses "intersection" mode so validation only rejects data missing fields that all metrics need. Metrics with optional fields that are absent at runtime are gracefully skipped via can_evaluate filtering in _evaluate.

Returns:

Type	Description
`set[str]`	set[str]: The required fields for the data.

`BaseEvaluator(name, metrics=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None)`

Bases: ABC

Base class for all evaluators.

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`required_fields`	`set[str]`	The required fields for the evaluator.
`input_type`	`type \| None`	The type of the input data.

Initialize the evaluator.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the evaluator.	required
`metrics`	`list[BaseMetric] \| None`	Metric instances configured for the evaluator. If None, no metrics are configured. Defaults to None.	`None`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If batch_status_check_interval or batch_max_iterations are not positive.

`aggregate_required_fields(metrics, mode='any')` `staticmethod`

Aggregate required fields from multiple metrics.

Parameters:

Name	Type	Description	Default
`metrics`	`Iterable[BaseMetric]`	The metrics to aggregate from.	required
`mode`	`str`	The aggregation mode. Options: - "union": All fields required by any metric - "intersection": Only fields required by all metrics - "any": Empty set (no validation) Defaults to "any".	`'any'`

Returns:

Type	Description
`set[str]`	set[str]: The aggregated required fields.

Raises:

Type	Description
`ValueError`	If mode is not one of the supported options.

`can_evaluate_any(metrics, data)` `staticmethod`

Check if any of the metrics can evaluate the given data.

Parameters:

Name	Type	Description	Default
`metrics`	`Iterable[BaseMetric]`	The metrics to check.	required
`data`	`LLMTestCase`	The data to validate against.	required

Returns:

Name	Type	Description
`bool`	`bool`	True if any metric can evaluate the data, False otherwise.

`ensure_list_of_dicts(data, key)` `staticmethod`

Ensure that a field in the data is a list of dictionaries.

Parameters:

Name	Type	Description	Default
`data`	`LLMTestCase`	The data to validate.	required
`key`	`str`	The key to check.	required

Raises:

Type	Description
`ValueError`	If the field is not a list or contains non-dictionary elements.

`ensure_non_empty_list(data, key)` `staticmethod`

Ensure that a field in the data is a non-empty list.

Parameters:

Name	Type	Description	Default
`data`	`LLMTestCase`	The data to validate.	required
`key`	`str`	The key to check.	required

Raises:

Type	Description
`ValueError`	If the field is not a list or is empty.

`evaluate(data)` `async`

Evaluate the data (single item or batch).

Parameters:

Name	Type	Description	Default
`data`	`LLMTestCase \| list[LLMTestCase]`	The data to be evaluated. Can be a single item or a list for batch processing.	required

Returns:

Type	Description
`EvaluatorResult \| list[EvaluatorResult]`	EvaluatorResult \| list[EvaluatorResult]: The evaluation output with aggregate_explanation. Returns a list if input is a list.

`get_input_fields()` `classmethod`

Return declared input field names if input_type is provided; otherwise None.

Returns:

Type	Description
`list[str] \| None`	list[str] \| None: The input fields.

`get_input_spec()` `classmethod`

Return structured spec for input fields if input_type is provided; otherwise None.

Returns:

Type	Description
`list[dict[str, Any]] \| None`	list[dict[str, Any]] \| None: The input spec.

`BaseGenerationEvaluator(models=None, metrics=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, refusal_metric=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

Bases: BaseEvaluator

Shared base evaluator for generation-style rule-engine evaluation.

Default expected input

input (str): The input provided to the AI system or component (e.g., a query, prompt, or instruction).
retrieved_context (str): Supporting context used during generation (e.g., retrieved documents).
expected_output (str): The reference output used for comparison.
actual_output (str): The output generated by the AI system or component to evaluate.

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`metrics`	`List[BaseMetric]`	The list of metrics to evaluate.
`run_parallel`	`bool`	Whether to run the metrics in parallel.
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation.
`metrics_aggregator`	`MetricsAggregator`	The aggregator for polarity-aware binary scoring.

Initialize the base generation evaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. `None` (default): single-judge mode using the default invoker. `[invoker]` or a single invoker: runs that invoker once. `[invoker] * N`: homogeneous — same model N times. `[invoker_a, invoker_b]`: heterogeneous — distinct models.	`None`
`metrics`	`list[BaseMetric] \| None`	Metric instances to evaluate. If None, default metrics are built.	`None`
`aggregation_method`	`AggregationSelector \| None`	The aggregation method to use for each metric.	`None`
`max_concurrent_judges`	`int \| None`	The maximum number of concurrent judges per metric.	`None`
`run_parallel`	`bool`	Whether to run the metrics in parallel. Defaults to True.	`True`
`refusal_metric`	`GEvalRefusalMetric \| None`	Optional explicit refusal metric.	`None`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	The aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every metric. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If models list contains invalid invoker configurations.

`ClassicalRetrievalEvaluator(metrics=None, k=20, metrics_aggregator=None)`

Bases: BaseEvaluator

A class that evaluates the performance of a classical retrieval system.

Required fields: - retrieved_chunks: The retrieved chunks with their similarity score. - ground_truth_chunk_ids: The ground truth chunk ids.

Example:

data = RetrievalData(
    retrieved_chunks={
        "chunk1": 0.9,
        "chunk2": 0.8,
        "chunk3": 0.7,
    },
    ground_truth_chunk_ids=["chunk1", "chunk2", "chunk3"],
)

evaluator = ClassicalRetrievalEvaluator()
await evaluator.evaluate(data)

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`metrics`	`list[str \| ClassicalRetrievalMetric] \| None`	The metrics to evaluate.
`k`	`int`	The number of retrieved chunks to consider.

Initializes the evaluator.

Parameters:

Name	Type	Description	Default
`metrics`	`list[str \| ClassicalRetrievalMetric] \| None`	The metrics to evaluate. Defaults to all metrics.	`None`
`k`	`int \| list[int]`	The number of retrieved chunks to consider. Defaults to 20.	`20`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`

`required_fields` `property`

Returns the required fields for the data.

Returns:

Type	Description
`set[str]`	set[str]: The required fields for the data.

`CompositeEvaluator(metrics, name='composite', parallel=True, metrics_aggregator=None)`

Bases: BaseEvaluator

Composite evaluator that runs multiple metrics and aggregates results.

This evaluator composes multiple BaseMetric objects and presents them as a single evaluator unit, following the GoF Composite pattern. It supports parallel or sequential execution, fault isolation, and custom aggregation.

Attributes:

Name	Type	Description
`metrics`	`list[BaseMetric]`	The list of metrics to evaluate.
`name`	`str`	The name of the evaluator.
`parallel`	`bool`	Whether to evaluate the metrics in parallel.
`metrics_aggregator`	`MetricsAggregator`	The aggregator for polarity-aware binary scoring.

Initialize the composite evaluator.

Parameters:

Name	Type	Description	Default
`metrics`	`list[BaseMetric]`	The list of metrics to evaluate.	required
`name`	`str`	The name of the evaluator.	`'composite'`
`parallel`	`bool`	Whether to evaluate the metrics in parallel.	`True`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`

`GEvalGenerationEvaluator(models=None, metrics=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, refusal_metric=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

Bases: BaseGenerationEvaluator

GEval Generation Evaluator.

This evaluator is used to evaluate the generation of the model.

Default expected input

input (str): The input provided to the AI system or component (e.g., a query, prompt, or instruction).
retrieved_context (str): Supporting context used during generation (e.g., retrieved documents).
expected_output (str): The reference output used for comparison.
actual_output (str): The output generated by the AI system or component to evaluate.

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`metrics`	`List[BaseMetric]`	The list of metrics to evaluate.
`run_parallel`	`bool`	Whether to run the metrics in parallel.
`metrics_aggregator`	`MetricsAggregator`	The aggregator for polarity-aware binary scoring.

Initialize the GEval Generation Evaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. `None` (default): single-judge mode using the default invoker. `[invoker]` or a single invoker: runs that invoker once. `[invoker] * N`: homogeneous — same model N times. `[invoker_a, invoker_b]`: heterogeneous — distinct models.	`None`
`metrics`	`list[BaseMetric] \| None`	Metric instances to evaluate. If None, uses `DEFAULT_METRICS`.	`None`
`aggregation_method`	`AggregationSelector \| None`	Strategy used to aggregate judge results. Defaults to None.	`None`
`max_concurrent_judges`	`int \| None`	Maximum number of judges to run concurrently. Defaults to None.	`None`
`run_parallel`	`bool`	Whether to run the metrics in parallel.	`True`
`refusal_metric`	`GEvalRefusalMetric \| None`	The refusal metric to use. If None, the default refusal metric will be used. Defaults to GEvalRefusalMetric.	`None`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	The aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every metric. Defaults to None.	`None`

`LMBasedRetrievalEvaluator(models=None, metrics=None, enabled_metrics=None, run_parallel=True, metrics_aggregator=None, fallback_models=None)`

Bases: BaseEvaluator

Evaluator for LM-based retrieval quality in RAG pipelines.

This evaluator

Runs a configurable set of retrieval metrics (by default: DeepEval contextual precision and contextual recall)
Combines their scores using a simple rule-based scheme to produce:
- relevancy_rating (good / bad / incomplete)
- score (aggregated retrieval score)

Default expected input

input (str): The input query to evaluate the metric.
expected_output (str): The expected output to evaluate the metric.
retrieved_context (str | list[str]): The list of retrieved contexts to evaluate the metric. If the retrieved context is a str, it will be converted into a list with a single element.

Attributes:

Name	Type	Description
`name`	`str`	The name of the evaluator.
`metrics`	`list[BaseMetric]`	The list of metrics to evaluate.
`enabled_metrics`	`Sequence[type[BaseMetric] \| str] \| None`	The list of metrics to enable.
`run_parallel`	`bool`	Whether to run the metrics in parallel.

Initialize the LM-based retrieval evaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. `None` (default): single-judge mode using the default invoker. `[invoker]` or a single invoker: runs that invoker once. `[invoker] * N`: homogeneous — same model N times. `[invoker_a, invoker_b]`: heterogeneous — distinct models.	`None`
`metrics`	`Sequence[BaseMetric] \| None`	Optional custom retrieval metric instances. If provided, these will be used as the base pool and may override the default metrics by name. Defaults to None.	`None`
`enabled_metrics`	`Sequence[type[BaseMetric] \| str] \| None`	Optional subset of metrics to enable from the metric pool. Each entry can be either a metric class or its `name`. If None, all metrics from the pool are used. Defaults to None.	`None`
`run_parallel`	`bool`	Whether to run retrieval metrics in parallel. Defaults to True.	`True`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every metric. Defaults to None.	`None`

`required_fields` `property`

Return the union of required fields from all configured metrics.

`QTEvaluator(models=None, completeness_metric=None, groundedness_metric=None, redundancy_metric=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, score_mapping=None, score_weights=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

Bases: BaseEvaluator

Evaluator for query transformation tasks.

Default expected input: - input (str): The query to evaluate the completeness of the model's output. - expected_output (str): The expected response to evaluate the completeness of the model's output. - actual_output (str): The generated response to evaluate the completeness of the model's output.

Attributes:

Name	Type	Description
`completeness_metric`	`GEvalCompletenessMetric`	The completeness metric.
`hallucination_metric`	`GEvalGroundednessMetric`	The groundedness metric.
`redundancy_metric`	`GEvalRedundancyMetric`	The redundancy metric.
`run_parallel`	`bool`	Whether to run the metrics in parallel.
`score_mapping`	`dict[str, dict[int, float]]`	The score mapping.
`score_weights`	`dict[str, float]`	The score weights.

Initialize the QTEvaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. `None` (default): single-judge mode using the default invoker. `[invoker]` or a single invoker: runs that invoker once. `[invoker] * N`: homogeneous — same model N times. `[invoker_a, invoker_b]`: heterogeneous — distinct models.	`None`
`completeness_metric`	`GEvalCompletenessMetric \| None`	The completeness metric. Defaults to built-in GEvalCompletenessMetric.	`None`
`groundedness_metric`	`GEvalGroundednessMetric \| None`	The groundedness metric. Defaults to built-in GEvalGroundednessMetric.	`None`
`redundancy_metric`	`GEvalRedundancyMetric \| None`	The redundancy metric. Defaults to built-in GEvalRedundancyMetric.	`None`
`aggregation_method`	`AggregationSelector \| None`	The aggregation method to use for each metric. If None, each metric uses its own default (MAJORITY_VOTE for GEval metrics).	`None`
`max_concurrent_judges`	`int \| None`	The maximum number of concurrent judges per metric. If None, each metric uses its own default.	`None`
`run_parallel`	`bool`	Whether to run the metrics in parallel. Defaults to True.	`True`
`score_mapping`	`dict[str, dict[int, float]] \| None`	The score mapping. Defaults to None. This is required if some of the default metrics are used.	`None`
`score_weights`	`dict[str, float] \| None`	The score weights. Defaults to None.	`None`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every metric. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If models list contains invalid invoker configurations.

`required_fields` `property`

Returns the required fields for the data.

Returns:

Type	Description
`set[str]`	set[str]: The required fields for the data.

`SummarizationEvaluator(models=None, metrics=None, aggregation_method=DefaultValues.AGGREGATION_METHOD, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

Bases: BaseGenerationEvaluator

Evaluator for summarization quality using four GEval-style metrics.

Default expected input

input (str): Source text or transcript.
actual_output (str): Generated summary.

Initialize the SummarizationEvaluator.

Parameters:

Name	Type	Description	Default
`models`	`BaseLMInvoker \| list[BaseLMInvoker] \| None`	Judge models for single-judge/multi-judge evaluation. `None` (default): single-judge mode using the default invoker. `[invoker]` or a single invoker: runs that invoker once. `[invoker] * N`: homogeneous — same model N times. `[invoker_a, invoker_b]`: heterogeneous — distinct models.	`None`
`metrics`	`list[BaseMetric] \| None`	Metric instances to evaluate. If None, uses `DEFAULT_METRICS`.	`None`
`aggregation_method`	`AggregationMethod`	Strategy used to aggregate judge results.	`AGGREGATION_METHOD`
`run_parallel`	`bool`	Whether to run the metrics in parallel. Defaults to True.	`True`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout.	`BATCH_MAX_ITERATIONS`
`metrics_aggregator`	`MetricsAggregator \| None`	Aggregator for polarity-aware binary scoring. If None, a default MetricsAggregator is used. Defaults to None.	`None`
`fallback_models`	`list[BaseLMInvoker] \| None`	Ordered fallback invoker chain propagated to every metric. Defaults to None.	`None`

Evaluator

AgentEvaluator(models=None, metrics=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)

Configure metrics — trajectory accuracy runs conditionally via can_evaluate

required_fields property

BaseEvaluator(name, metrics=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None)

aggregate_required_fields(metrics, mode='any') staticmethod

can_evaluate_any(metrics, data) staticmethod

ensure_list_of_dicts(data, key) staticmethod

ensure_non_empty_list(data, key) staticmethod

evaluate(data) async

get_input_fields() classmethod

get_input_spec() classmethod

ClassicalRetrievalEvaluator(metrics=None, k=20, metrics_aggregator=None)

required_fields property

CompositeEvaluator(metrics, name='composite', parallel=True, metrics_aggregator=None)

LMBasedRetrievalEvaluator(models=None, metrics=None, enabled_metrics=None, run_parallel=True, metrics_aggregator=None, fallback_models=None)

required_fields property

required_fields property

SummarizationEvaluator(models=None, metrics=None, aggregation_method=DefaultValues.AGGREGATION_METHOD, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)

`AgentEvaluator(models=None, metrics=None, aggregation_method=None, max_concurrent_judges=None, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`

`required_fields` `property`

`BaseEvaluator(name, metrics=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None)`

`aggregate_required_fields(metrics, mode='any')` `staticmethod`

`can_evaluate_any(metrics, data)` `staticmethod`

`ensure_list_of_dicts(data, key)` `staticmethod`

`ensure_non_empty_list(data, key)` `staticmethod`

`evaluate(data)` `async`

`get_input_fields()` `classmethod`

`get_input_spec()` `classmethod`

`ClassicalRetrievalEvaluator(metrics=None, k=20, metrics_aggregator=None)`

`required_fields` `property`

`CompositeEvaluator(metrics, name='composite', parallel=True, metrics_aggregator=None)`

`LMBasedRetrievalEvaluator(models=None, metrics=None, enabled_metrics=None, run_parallel=True, metrics_aggregator=None, fallback_models=None)`

`required_fields` `property`

`required_fields` `property`

`SummarizationEvaluator(models=None, metrics=None, aggregation_method=DefaultValues.AGGREGATION_METHOD, run_parallel=True, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, metrics_aggregator=None, fallback_models=None)`