Evaluator

Evaluator module for orchestrating evaluation workflows.

This module provides evaluator classes that coordinate metrics and evaluation logic for different use cases. Evaluators handle the evaluation process, including data preparation, metric execution, and result aggregation.

Available evaluators: - BaseEvaluator: Abstract base class for all evaluators - GenerationEvaluator: Evaluates text generation quality - AgentEvaluator: Evaluates AI agent performance - ClassicalRetrievalEvaluator: Traditional retrieval evaluation methods - LMBasedRetrievalEvaluator: LM-based retrieval evaluation methods - GEvalGenerationEvaluator: G-Eval based generation evaluation - RAGEvaluator: Combined retrieval and generation evaluation for RAG pipelines - QTEvaluator: Question-answering evaluation - CustomEvaluator: Create custom evaluation workflows

AgentEvaluator(model=DefaultValues.AGENT_EVALS_MODEL, model_credentials=None, model_config=None, prompt=None, use_reference=True, continuous=False, choices=None, use_reasoning=True, few_shot_examples=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)

Bases: BaseEvaluator

Evaluator for agent tasks.

This evaluator uses the LangChain AgentEvals trajectory accuracy metric to evaluate the performance of AI agents based on their execution trajectories.

Default expected input

• agent_trajectory (list[dict[str, Any]]): The agent trajectory containing the sequence of actions, tool calls, and responses.
• expected_agent_trajectory (list[dict[str, Any]] | None, optional): The expected agent trajectory for reference-based evaluation.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
trajectory_accuracy_metric	LangChainAgentTrajectoryAccuracyMetric	The metric used to evaluate agent trajectory accuracy.

Initialize the AgentEvaluator.

Parameters:

Name	Type	Description	Default
model	str | ModelId | BaseLMInvoker	The model to use for the trajectory accuracy metric. Defaults to DefaultValues.AGENT_EVALS_MODEL.	AGENT_EVALS_MODEL
model_credentials	str | None	The model credentials. Defaults to None. This is required for the metric to function properly.	None
model_config	dict[str, Any] | None	The model configuration. Defaults to None.	None
prompt	str | None	Custom prompt for evaluation. If None, uses the default prompt from the metric. Defaults to None.	None
use_reference	bool	Whether to use expected_agent_trajectory for reference-based evaluation. Defaults to True.	True
continuous	bool	If True, score will be a float between 0 and 1. If False, score will be boolean. Defaults to False.	False
choices	list[float] | None	Optional list of specific float values the score must be chosen from. Defaults to None.	None
use_reasoning	bool	If True, includes explanation for the score in the output. Defaults to True.	True
few_shot_examples	list[Any] | None	Optional list of example evaluations to append to the prompt. Defaults to None.	None
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS

Raises:

Type	Description
ValueError	If model_credentials is not provided.

required_fields property

Returns the required fields for the data.

Returns:

Type	Description
set[str]	set[str]: The required fields for the data.

BaseEvaluator(name, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)

Bases: ABC

Base class for all evaluators.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
required_fields	set[str]	The required fields for the evaluator.
input_type	type | None	The type of the input data.

Initialize the evaluator.

Parameters:

Name	Type	Description	Default
name	str	The name of the evaluator.	required
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS

Raises:

Type	Description
ValueError	If batch_status_check_interval or batch_max_iterations are not positive.

aggregate_required_fields(metrics, mode='any') staticmethod

Aggregate required fields from multiple metrics.

Parameters:

Name	Type	Description	Default
metrics	Iterable[BaseMetric]	The metrics to aggregate from.	required
mode	str	The aggregation mode. Options: - "union": All fields required by any metric - "intersection": Only fields required by all metrics - "any": Empty set (no validation) Defaults to "any".	'any'

Returns:

Type	Description
set[str]	set[str]: The aggregated required fields.

Raises:

Type	Description
ValueError	If mode is not one of the supported options.

can_evaluate_any(metrics, data) staticmethod

Check if any of the metrics can evaluate the given data.

Parameters:

Name	Type	Description	Default
metrics	Iterable[BaseMetric]	The metrics to check.	required
data	MetricInput	The data to validate against.	required

Returns:

Name	Type	Description
bool	bool	True if any metric can evaluate the data, False otherwise.

ensure_list_of_dicts(data, key) staticmethod

Ensure that a field in the data is a list of dictionaries.

Parameters:

Name	Type	Description	Default
data	MetricInput	The data to validate.	required
key	str	The key to check.	required

Raises:

Type	Description
ValueError	If the field is not a list or contains non-dictionary elements.

ensure_non_empty_list(data, key) staticmethod

Ensure that a field in the data is a non-empty list.

Parameters:

Name	Type	Description	Default
data	MetricInput	The data to validate.	required
key	str	The key to check.	required

Raises:

Type	Description
ValueError	If the field is not a list or is empty.

evaluate(data) async

Evaluate the data (single item or batch).

Parameters:

Name	Type	Description	Default
data	MetricInput | list[MetricInput]	The data to be evaluated. Can be a single item or a list for batch processing.	required

Returns:

Type	Description
EvaluationOutput | list[EvaluationOutput]	EvaluationOutput | list[EvaluationOutput]: The evaluation output with global_explanation. Returns a list if input is a list.

get_input_fields() classmethod

Return declared input field names if input_type is provided; otherwise None.

Returns:

Type	Description
list[str] | None	list[str] | None: The input fields.

get_input_spec() classmethod

Return structured spec for input fields if input_type is provided; otherwise None.

Returns:

Type	Description
list[dict[str, Any]] | None	list[dict[str, Any]] | None: The input spec.

ClassicalRetrievalEvaluator(metrics=None, k=20)

Bases: BaseEvaluator

A class that evaluates the performance of a classical retrieval system.

Required fields: - retrieved_chunks: The retrieved chunks with their similarity score. - ground_truth_chunk_ids: The ground truth chunk ids.

Example:

data = RetrievalData(
    retrieved_chunks={
        "chunk1": 0.9,
        "chunk2": 0.8,
        "chunk3": 0.7,
    },
    ground_truth_chunk_ids=["chunk1", "chunk2", "chunk3"],
)

evaluator = ClassicalRetrievalEvaluator()
await evaluator.evaluate(data)

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	list[str | ClassicalRetrievalMetric] | None	The metrics to evaluate.
k	int	The number of retrieved chunks to consider.

Initializes the evaluator.

Parameters:

Name	Type	Description	Default
metrics	list[str | ClassicalRetrievalMetric] | None	The metrics to evaluate. Defaults to all metrics.	None
k	int | list[int]	The number of retrieved chunks to consider. Defaults to 20.	20

required_fields property

Returns the required fields for the data.

Returns:

Type	Description
set[str]	set[str]: The required fields for the data.

CustomEvaluator(metrics, name='custom', parallel=True)

Bases: BaseEvaluator

Custom evaluator.

This evaluator is used to evaluate the performance of the model.

Attributes:

Name	Type	Description
metrics	list[BaseMetric]	The list of metrics to evaluate.
name	str	The name of the evaluator.
parallel	bool	Whether to evaluate the metrics in parallel.

Initialize the custom evaluator.

Parameters:

Name	Type	Description	Default
metrics	list[BaseMetric]	The list of metrics to evaluate.	required
name	str	The name of the evaluator.	'custom'
parallel	bool	Whether to evaluate the metrics in parallel.	True

GEvalGenerationEvaluator(metrics=None, enabled_metrics=None, model=DefaultValues.MODEL, model_credentials=None, model_config=None, run_parallel=True, rule_book=None, generation_rule_engine=None, judge=None, refusal_metric=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)

Bases: GenerationEvaluator

GEval Generation Evaluator.

This evaluator is used to evaluate the generation of the model.

Default expected input

• query (str): The query to evaluate the generation of the model's output.
• retrieved_context (str): The retrieved context to evaluate the generation of the model's output.
• expected_response (str): The expected response to evaluate the generation of the model's output.
• generated_response (str): The generated response to evaluate the generation of the model's output.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	List[BaseMetric]	The list of metrics to evaluate.
run_parallel	bool	Whether to run the metrics in parallel.
rule_book	RuleBook | None	The rule book.
generation_rule_engine	GenerationRuleEngine | None	The generation rule engine.

Initialize the GEval Generation Evaluator.

Parameters:

Name	Type	Description	Default
metrics	List[BaseMetric] | None	The list of metrics to evaluate.	None
enabled_metrics	List[type[BaseMetric] | str] | None	The list of enabled metrics.	None
model	str | ModelId | BaseLMInvoker	The model to use for the metrics.	MODEL
model_credentials	str | None	The model credentials to use for the metrics.	None
model_config	dict[str, Any] | None	The model config to use for the metrics.	None
run_parallel	bool	Whether to run the metrics in parallel.	True
rule_book	RuleBook | None	The rule book.	None
generation_rule_engine	GenerationRuleEngine | None	The generation rule engine.	None
judge	MultipleLLMAsJudge | None	Optional multiple LLM judge for ensemble evaluation.	None
refusal_metric	type[BaseMetric] | None	The refusal metric to use. If None, the default refusal metric will be used. Defaults to GEvalRefusalMetric.	None
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS

GenerationEvaluator(metrics=None, enabled_metrics=None, model=DefaultValues.MODEL, model_credentials=None, model_config=None, run_parallel=True, rule_book=None, generation_rule_engine=None, judge=None, refusal_metric=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)

Bases: BaseEvaluator

Evaluator for generation tasks.

Default expected input

• query (str): The query to evaluate the completeness of the model's output.
• retrieved_context (str): The retrieved context to evaluate the groundedness of the model's output.
• expected_response (str): The expected response to evaluate the completeness of the model's output.
• generated_response (str): The generated response to evaluate the completeness of the model's output.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	List[BaseMetric]	The list of metrics to evaluate.
run_parallel	bool	Whether to run the metrics in parallel.
rule_book	RuleBook | None	The rule book.
generation_rule_engine	GenerationRuleEngine | None	The generation rule engine.
judge	MultipleLLMAsJudge | None	Optional multiple LLM judge for ensemble evaluation.

Initialize the GenerationEvaluator.

Parameters:

Name	Type	Description	Default
metrics	List[BaseMetric] | None	A list of metric instances to use as a base pool. If None, defaults to [CompletenessMetric, RedundancyMetric, GroundednessMetric]. Each custom metrics, must generate a score key in the output.	None
enabled_metrics	List[type[BaseMetric] | str] | None	A list of metric classes or names to enable from the pool. If None, all metrics in the pool are used.	None
model	str | ModelId | BaseLMInvoker	The model to use for the metrics.	MODEL
model_config	dict[str, Any] | None	The model config to use for the metrics.	None
model_credentials	str | None	The model credentials, used for initializing default metrics. Defaults to None. This is required if some of the default metrics are used.	None
run_parallel	bool	Whether to run the metrics in parallel. Defaults to False.	True
rule_book	RuleBook | None	The rule book for evaluation. If not provided, a default one is generated, but only if all enabled metrics are from the default set.	None
generation_rule_engine	GenerationRuleEngine | None	The generation rule engine. Defaults to a new instance with the determined rule book.	None
judge	MultipleLLMAsJudge | None	Optional multiple LLM judge for ensemble evaluation. If provided, will use multiple judges instead of single model evaluation. Uses composition pattern for clean separation of concerns.	None
refusal_metric	type[RefusalMetric] | None	The refusal metric to use. If None, the default refusal metric will be used. Defaults to RefusalMetric.	None
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS

Raises:

Type	Description
ValueError	If model_credentials is not provided when using default metrics.
ValueError	If a custom rule_book is provided when using custom metrics or a mix of custom and default metrics without an explicit rule_book.

HybridRuleBook(good, bad, issue_rules) dataclass

Rule book that supports both MetricSpec (int) and FloatMetricSpec (float).

This allows combining generation metrics (int-based) and retrieval metrics (float-based) in a single rule book.

Attributes:

Name	Type	Description
good	Specification	The good rule (can be MetricSpec, FloatMetricSpec, or combination).
bad	Specification	The bad rule (can be MetricSpec, FloatMetricSpec, or combination).
issue_rules	Mapping[Issue, Specification]	Issue detection rules.

HybridRuleEngine(rules)

Bases: BaseRuleEngine[HybridRuleBook, Specification, Relevancy]

Rule engine that handles both int-based (MetricSpec) and float-based (FloatMetricSpec) metrics.

This engine can evaluate rules that combine generation metrics (int 0-4) and retrieval metrics (float 0.0-1.0) in a single rule book.

Initialize the HybridRuleEngine.

LMBasedRetrievalEvaluator(metrics=None, enabled_metrics=None, model=DefaultValues.MODEL, model_credentials=None, model_config=None, run_parallel=True, rule_book=None, rule_engine=None)

Bases: BaseEvaluator

Evaluator for LM-based retrieval quality in RAG pipelines.

This evaluator

• Runs a configurable set of retrieval metrics (by default: DeepEval contextual precision and contextual recall)
• Combines their scores using a simple rule-based scheme to produce:
  • relevancy_rating (good / bad / incomplete)
  • score (aggregated retrieval score)
  • possible_issues (list of textual issues)

Default expected input

• query (str): The query to evaluate the metric.
• expected_response (str): The expected response to evaluate the metric.
• retrieved_context (str | list[str]): The list of retrieved contexts to evaluate the metric. If the retrieved context is a str, it will be converted into a list with a single element.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	list[BaseMetric]	The list of metrics to evaluate.
enabled_metrics	Sequence[type[BaseMetric] | str] | None	The list of metrics to enable.
model	str | ModelId | BaseLMInvoker	The model to use for the metrics.
model_credentials	str | None	The model credentials to use for the metrics.
model_config	dict[str, Any] | None	The model configuration to use for the metrics.
run_parallel	bool	Whether to run the metrics in parallel.
rule_book	LMBasedRetrievalRuleBook | None	The rule book for evaluation.
rule_engine	LMBasedRetrievalRuleEngine | None	The rule engine for classification.

Initialize the LM-based retrieval evaluator.

Parameters:

Name	Type	Description	Default
metrics	Sequence[BaseMetric] | None	Optional custom retrieval metric instances. If provided, these will be used as the base pool and may override the default metrics by name. Defaults to None.	None
enabled_metrics	Sequence[type[BaseMetric] | str] | None	Optional subset of metrics to enable from the metric pool. Each entry can be either a metric class or its name. If None, all metrics from the pool are used. Defaults to None.	None
model	str | ModelId | BaseLMInvoker	Model for the default DeepEval metrics. Defaults to DefaultValues.MODEL.	MODEL
model_credentials	str | None	Credentials for the model, required when model is a string. Defaults to None.	None
model_config	dict[str, Any] | None	Optional model configuration. Defaults to None.	None
run_parallel	bool	Whether to run retrieval metrics in parallel. Defaults to True.	True
rule_book	LMBasedRetrievalRuleBook | None	The rule book for evaluation. If not provided, a default one is generated based on enabled metrics. Defaults to None.	None
rule_engine	LMBasedRetrievalRuleEngine | None	The rule engine for classification. If not provided, a new instance is created with the determined rule book. Defaults to None.	None

required_fields property

Return the union of required fields from all configured metrics.

QTEvaluator(completeness_metric=None, groundedness_metric=None, redundancy_metric=None, model=DefaultValues.MODEL, model_config=None, model_credentials=None, run_parallel=True, score_mapping=None, score_weights=None, judge=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)

Bases: BaseEvaluator

Evaluator for query transformation tasks.

Default expected input: - query (str): The query to evaluate the completeness of the model's output. - expected_response (str): The expected response to evaluate the completeness of the model's output. - generated_response (str): The generated response to evaluate the completeness of the model's output.

Attributes:

Name	Type	Description
completeness_metric	CompletenessMetric	The completeness metric.
hallucination_metric	GroundednessMetric	The groundedness metric.
redundancy_metric	RedundancyMetric	The redundancy metric.
run_parallel	bool	Whether to run the metrics in parallel.
score_mapping	dict[str, dict[int, float]]	The score mapping.
score_weights	dict[str, float]	The score weights.

Initialize the QTEvaluator.

Parameters:

Name	Type	Description	Default
completeness_metric	CompletenessMetric | None	The completeness metric. Defaults to built-in CompletenessMetric.	None
groundedness_metric	GroundednessMetric | None	The groundedness metric. Defaults to built-in GroundednessMetric.	None
redundancy_metric	RedundancyMetric | None	The redundancy metric. Defaults to built-in RedundancyMetric.	None
model	str | ModelId | BaseLMInvoker	The model to use for the metrics.	MODEL
model_config	dict[str, Any] | None	The model config to use for the metrics.	None
model_credentials	str | None	The model credentials. Defaults to None. This is required if some of the default metrics are used.	None
run_parallel	bool	Whether to run the metrics in parallel. Defaults to True.	True
score_mapping	dict[str, dict[int, float]] | None	The score mapping. Defaults to None. This is required if some of the default metrics are used.	None
score_weights	dict[str, float] | None	The score weights. Defaults to None.	None
judge	MultipleLLMAsJudge | None	Optional multiple LLM judge for ensemble evaluation.	None
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120 (60 minutes with default interval).	BATCH_MAX_ITERATIONS

Raises:

Type	Description
ValueError	If model_credentials is not provided when using default metrics.

required_fields property

Returns the required fields for the data.

Returns:

Type	Description
set[str]	set[str]: The required fields for the data.

RAGEvaluator(retrieval_evaluator=None, generation_evaluator=None, retrieval_metrics=None, generation_metrics=None, model=DefaultValues.MODEL, model_credentials=None, model_config=None, run_parallel=True, rule_book=None, rule_engine=None, judge=None, refusal_metric=None)

Bases: BaseEvaluator

Evaluator for RAG pipelines combining retrieval and generation evaluation.

This evaluator

• Runs retrieval evaluation using LMBasedRetrievalEvaluator
• Runs generation evaluation using GEvalGenerationEvaluator
• Combines their scores using a customizable rule-based scheme to produce:
  • relevancy_rating (good / bad / incomplete)
  • score (aggregated RAG score)
  • possible_issues (list of textual issues)

Important Note on Rule Engine: By default, this evaluator uses GenerationRuleEngine with RuleBook which works with generation metrics (int scores 0-4). To include retrieval metrics in rule-based classification, use HybridRuleBook with HybridRuleEngine, which supports both MetricSpec (int-based) and FloatMetricSpec (float-based) metrics.

Default expected input

• query (str): The query to evaluate.
• expected_response (str): The expected response.
• retrieved_context (str | list[str]): The retrieved contexts.
• generated_response (str): The generated response.

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
retrieval_evaluator	LMBasedRetrievalEvaluator	The retrieval evaluator.
generation_evaluator	GEvalGenerationEvaluator	The generation evaluator.
rule_book	RuleBook | HybridRuleBook | None	The rule book for evaluation (uses generation metrics by default). Use HybridRuleBook to combine both generation (int) and retrieval (float) metrics.
rule_engine	GenerationRuleEngine | HybridRuleEngine | None	The rule engine for classification. Uses GenerationRuleEngine for RuleBook or HybridRuleEngine for HybridRuleBook.
run_parallel	bool	Whether to run retrieval and generation evaluations in parallel.

Initialize the RAG evaluator.

Parameters:

Name	Type	Description	Default
retrieval_evaluator	LMBasedRetrievalEvaluator | None	Pre-configured retrieval evaluator. If provided, this will be used directly and retrieval_* parameters will be ignored. Defaults to None.	None
generation_evaluator	GEvalGenerationEvaluator | None	Pre-configured generation evaluator. If provided, this will be used directly and generation_* parameters will be ignored. Defaults to None.	None
retrieval_metrics	Sequence[BaseMetric] | None	Optional custom retrieval metric instances. Used only if retrieval_evaluator is None. If provided, these will be used as the base pool and may override the default metrics by name. Defaults to None.	None
generation_metrics	Sequence[BaseMetric] | None	Optional custom generation metric instances. Used only if generation_evaluator is None. If provided, these will be used as the base pool and may override the default metrics by name. Defaults to None.	None
model	str | ModelId | BaseLMInvoker	Model for the default metrics. Used only if evaluators are None. Defaults to DefaultValues.MODEL.	MODEL
model_credentials	str | None	Credentials for the model, required when model is a string. Used only if evaluators are None. Defaults to None.	None
model_config	dict[str, Any] | None	Optional model configuration. Used only if evaluators are None. Defaults to None.	None
run_parallel	bool	Whether to run retrieval and generation evaluations in parallel. Used only if evaluators are None. Defaults to True.	True
rule_book	RuleBook | HybridRuleBook | None	The rule book for evaluation. If not provided, a default one is generated based on enabled generation metrics. Use RuleBook for generation-only metrics (int-based) or HybridRuleBook to combine both generation (int) and retrieval (float) metrics. Defaults to None.	None
rule_engine	GenerationRuleEngine | HybridRuleEngine | None	The rule engine for classification. If not provided, a new instance is created with the determined rule book. Use GenerationRuleEngine for RuleBook or HybridRuleEngine for HybridRuleBook. Defaults to None.	None
judge	MultipleLLMAsJudge | None	Optional multiple LLM judge for ensemble evaluation. Used only if generation_evaluator is None. Defaults to None.	None
refusal_metric	type[BaseMetric] | None	The refusal metric to use for generation evaluator. Used only if generation_evaluator is None. If None, the default refusal metric will be used. Defaults to None.	None

required_fields property

Return the union of required fields from both evaluators.

TrajectoryGenerationEvaluator(agent_evaluator=None, generation_evaluator=None)

Bases: BaseEvaluator

Evaluator for agent trajectory and generation quality evaluation with rule-based aggregation.

This evaluator combines: 1. LangChain AgentEvals trajectory accuracy metric (agent execution quality) 2. GEval generation evaluator (completeness, groundedness, redundancy) 3. Rule-based aggregation logic

Uses dependency injection pattern - accepts pre-configured evaluators for maximum flexibility.

Expected input (AgentData with generation fields): - agent_trajectory (list[dict[str, Any]]): The agent trajectory - expected_agent_trajectory (list[dict[str, Any]] | None, optional): Expected trajectory - query (str | None, optional): The query for generation evaluation - generated_response (str | None, optional): The generated response - expected_response (str | None, optional): The expected response - retrieved_context (str | list[str] | None, optional): The retrieved context

Aggregation Logic

• If trajectory relevancy is "incomplete" or "bad": return trajectory result
• If trajectory relevancy is "good": return GEval generation result

Output Structure:

The output is a flat dictionary containing:

- Aggregated Results:
    - global_explanation (str): Human-readable explanation of the evaluation
    (added by parent evaluator)
    - score (float | int): Final aggregated score based on rule-based logic
    - relevancy_rating (str): Final relevancy rating ("good", "bad", or "incomplete")
    - possible_issues (list[str], optional): List of detected issues
    (only present when trajectory is good)

- Trajectory Evaluation Results:
    - langchain_agent_trajectory_accuracy (dict): Trajectory evaluation result containing
    score, explanation, key, and metadata from the agent trajectory metric

- Generation Evaluation Results (nested):
    - generation (dict): Generation evaluation results containing:
        - global_explanation (str): Explanation of generation quality
        - relevancy_rating (str): Generation quality rating
        - score (float | int): Generation quality score
        - possible_issues (list[str]): List of generation-related issues
        - Individual metric results (completeness, groundedness, redundancy,
        language_consistency, refusal_alignment)

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
agent_evaluator	AgentEvaluator	Pre-configured evaluator for trajectory assessment.
generation_evaluator	GEvalGenerationEvaluator	Pre-configured evaluator for generation quality assessment.

Example

import os from gllm_evals.evaluator.agent_evaluator import AgentEvaluator from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator

Configure individual evaluators

agent_eval = AgentEvaluator( ... model="openai/gpt-4o-mini", ... model_credentials=os.getenv("OPENAI_API_KEY"), ... use_reference=True, ... ) gen_eval = GEvalGenerationEvaluator( ... model="google/gemini-2.0-flash", ... model_credentials=os.getenv("GOOGLE_API_KEY"), ... run_parallel=True, ... )

Create combined evaluator

evaluator = TrajectoryGenerationEvaluator( ... agent_evaluator=agent_eval, ... generation_evaluator=gen_eval, ... )

Evaluate

result = await evaluator.evaluate(data)

Initialize the TrajectoryGenerationEvaluator with optional dependency injection.

Parameters:

Name	Type	Description	Default
agent_evaluator	AgentEvaluator | None	Pre-configured evaluator for agent trajectory evaluation. If None, a default AgentEvaluator will be created with model=DefaultValues.AGENT_EVALS_MODEL. Defaults to None.	None
generation_evaluator	GEvalGenerationEvaluator | None	Pre-configured evaluator for generation quality assessment. If None, a default GEvalGenerationEvaluator will be created with model=DefaultValues.MODEL. Defaults to None.	None

Note

When using default evaluators (by not providing agent_evaluator or generation_evaluator), make sure the required environment variables are set: - OPENAI_API_KEY for agent evaluator - GOOGLE_API_KEY for generation evaluator

required_fields property

Returns the required fields for the data.

Returns the combined set of required fields from both the trajectory evaluator and generation evaluator.

Returns:

Type	Description
set[str]	set[str]: The required fields for the data.