Langchain openevals

This module contains the LangChainOpenEvalsMetric and LangChainOpenEvalsLLMAsAJudgeMetric.

Authors

Surya Mahadi (made.r.s.mahadi@gdplabs.id)

References

[1] https://github.com/langchain-ai/openevals

LangChainOpenEvalsLLMAsAJudgeMetric(name, prompt, model, system=None, credentials=None, config=None, schema=None, feedback_key='score', continuous=False, choices=None, use_reasoning=True, few_shot_examples=None)

Bases: LangChainOpenEvalsMetric

A metric that uses LangChain and OpenEvals to evaluate the LLM as a judge.

Attributes:

Name	Type	Description
name	str	The name of the metric.
prompt	str	The prompt to use.
model	str | ModelId | BaseLMInvoker	The model to use.

Initialize the LangChainOpenEvalsLLMAsAJudgeMetric.

Parameters:

Name	Type	Description	Default
name	str	The name of the metric.	required
prompt	str	The evaluation prompt, can be a string template, LangChain prompt template, or callable that returns a list of chat messages.	required
model	str | ModelId | BaseLMInvoker	The model to use.	required
system	str | None	Optional system message to prepend to the prompt.	None
credentials	str | None	The credentials to use for the model. Defaults to None.	None
config	dict[str, Any] | None	The config to use for the model. Defaults to None.	None
schema	ResponseSchema | None	The schema to use for the model. Defaults to None.	None
feedback_key	str	Key used to store the evaluation result, defaults to "score".	'score'
continuous	bool	If True, score will be a float between 0 and 1. If False, score will be boolean. Defaults to False.	False
choices	list[float] | None	Optional list of specific float values the score must be chosen from. Defaults to None.	None
use_reasoning	bool	If True, includes explanation for the score in the output. Defaults to True.	True
few_shot_examples	list[FewShotExample] | None	Optional list of example evaluations to append to the prompt. Defaults to None.	None

LangChainOpenEvalsMetric(name, evaluator)

Bases: BaseMetric

A metric that uses LangChain and OpenEvals.

Attributes:

Name	Type	Description
name	str	The name of the metric.
evaluator	Union[SimpleAsyncEvaluator, Callable[..., Awaitable[Any]]]	The evaluator to use.

Initialize the LangChainOpenEvalsMetric.

Parameters:

Name	Type	Description	Default
name	str	The name of the metric.	required
evaluator	Union[SimpleAsyncEvaluator, Callable[..., Awaitable[Any]]]	The evaluator to use.	required