Classical retrieval evaluator

Classical retrieval evaluator.

Authors

Surya Mahadi (made.r.s.mahadi@gdplabs.id)

References

None

ClassicalRetrievalEvaluator(metrics=None, k=20)

Bases: BaseEvaluator

A class that evaluates the performance of a classical retrieval system.

Required fields: - retrieved_chunks: The retrieved chunks with their similarity score. - ground_truth_chunk_ids: The ground truth chunk ids.

Example:

data = RetrievalData(
    retrieved_chunks={
        "chunk1": 0.9,
        "chunk2": 0.8,
        "chunk3": 0.7,
    },
    ground_truth_chunk_ids=["chunk1", "chunk2", "chunk3"],
)

evaluator = ClassicalRetrievalEvaluator()
await evaluator.evaluate(data)

Attributes:

Name	Type	Description
name	str	The name of the evaluator.
metrics	list[str | ClassicalRetrievalMetric] | None	The metrics to evaluate.
k	int	The number of retrieved chunks to consider.

Initializes the evaluator.

Parameters:

Name	Type	Description	Default
metrics	list[str | ClassicalRetrievalMetric] | None	The metrics to evaluate. Defaults to all metrics.	None
k	int | list[int]	The number of retrieved chunks to consider. Defaults to 20.	20

required_fields: set[str] property

Returns the required fields for the data.

Returns:

Type	Description
set[str]	set[str]: The required fields for the data.

ClassicalRetrievalMetric

Bases: StrEnum

A list of metrics that can be used to evaluate the performance of a classical retrieval system.

Attributes:

Name	Type	Description
MAP		Mean Average Precision.
NDCG		Normalized Discounted Cumulative Gain.
PRECISION		Precision.
RECALL		Recall.
TOP_K_ACCURACY		Top K Accuracy.

all() classmethod

Returns all the metrics in the enum.

Returns:

Type	Description
list[str]	list[str]: A list of all the metrics in the enum.