Deepeval

DeepEval Metric Integration.

Authors

Surya Mahadi (made.r.s.mahadi@gdplabs.id)

References

[1] https://deepeval.com/docs/getting-started

DeepEvalMetric(metric, name)

Bases: BaseMetric

DeepEval Metric Integration.

Attributes:

Name	Type	Description
metric	BaseMetric	The DeepEval metric to wrap.
name	str	The name of the metric.

Available Fields: - query (str): The query to evaluate the metric. Similar to input in LLMTestCaseParams. - generated_response (str | list[str], optional): The generated response to evaluate the metric. Similar to actual_output in LLMTestCaseParams. If the generated response is a list, the responses are concatenated into a single string. - expected_response (str | list[str], optional): The expected response to evaluate the metric. Similar to expected_output in LLMTestCaseParams. If the expected response is a list, the responses are concatenated into a single string. - expected_retrieved_context (str | list[str], optional): The expected retrieved context to evaluate the metric. Similar to context in LLMTestCaseParams. If the expected retrieved context is a str, it will be converted into a list with a single element. - retrieved_context (str | list[str], optional): The list of retrieved contexts to evaluate the metric. Similar to retrieval_context in LLMTestCaseParams. If the retrieved context is a str, it will be converted into a list with a single element.

Initializes the DeepEvalMetric class.

Parameters:

Name	Type	Description	Default
metric	BaseMetric	The DeepEval metric to wrap.	required
name	str	The name of the metric.	required

DeepEvalMetricFactory(name, model, model_credentials, model_config, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS, **kwargs)

Bases: DeepEvalMetric, ABC

Abstract base class for creating DeepEval metrics with a shared model invoker.

Initializes the metric, handling common model invoker creation.

Parameters:

Name	Type	Description	Default
name	str	The name for the metric.	required
model	Union[str, ModelId, BaseLMInvoker]	The model identifier or an existing LM invoker instance.	required
model_credentials	Optional[str]	Credentials for the model, required if model is a string.	required
model_config	Optional[Dict[str, Any]]	Configuration for the model.	required
batch_status_check_interval	float	Time between batch status checks in seconds. Defaults to 30.0.	BATCH_STATUS_CHECK_INTERVAL
batch_max_iterations	int	Maximum number of status check iterations before timeout. Defaults to 120.	BATCH_MAX_ITERATIONS
**kwargs		Additional arguments for the specific DeepEval metric constructor.	{}