Refusal

Refusal metric.

This metric is used to evaluate the refusal of the model's output.

`RefusalMetric(model=DefaultValues.MODEL, model_credentials=None, model_config=None, prompt_builder=None, response_schema=None, batch_status_check_interval=DefaultValues.BATCH_STATUS_CHECK_INTERVAL, batch_max_iterations=DefaultValues.BATCH_MAX_ITERATIONS)`

Bases: LMBasedMetric

Refusal metric.

This metric is used to evaluate the refusal of the model's output.

Available Fields

query (str): The query.
expected_response (str): The expected response.

Scoring

0-1 (Categorical): 0 means not refusal, 1 means refusal.

Cookbook Example

Please refer to example_refusal.py in the gen-ai-sdk-cookbook repository.

Initialize the RefusalMetric class.

Default expected input: - query (str): The query to evaluate the refusal of the model's output. - expected_response (str): The expected response to evaluate the refusal of the model's output.

Parameters:

Name	Type	Description	Default
`model`	`Union[str, ModelId, BaseLMInvoker]`	The model to use for the metric.	`MODEL`
`model_credentials`	`str \| None`	The model credentials to use for the metric. Defaults to None.	`None`
`model_config`	`dict[str, Any] \| None`	The model config to use for the metric. Defaults to an empty dictionary.	`None`
`prompt_builder`	`PromptBuilder \| None`	The prompt builder to use for the metric. Defaults to default prompt builder.	`None`
`response_schema`	`ResponseSchema \| None`	The response schema to use for the metric. Defaults to RefusalResponseSchema.	`None`
`batch_status_check_interval`	`float`	Time between batch status checks in seconds. Defaults to 30.0.	`BATCH_STATUS_CHECK_INTERVAL`
`batch_max_iterations`	`int`	Maximum number of status check iterations before timeout. Defaults to 120.	`BATCH_MAX_ITERATIONS`

`RefusalResponseSchema`

Bases: BaseModel

Response schema for the refusal metric.

Attributes:

Name	Type	Description
`question`	`str`	The question that was asked.
`expected_output_statements`	`list[str]`	The expected output statements.
`count`	`str`	The count of the generated output statements.
`score`	`int`	The score of the generated output statements.
`explanation`	`str`	The explanation of the generated output statements.