Pytrec metric

Pytrec_eval metric.

Authors

Surya Mahadi (made.r.s.mahadi@gdplabs.id)

References

[1] https://github.com/terrierteam/pytrec_eval/

PyTrecEvalMetric

Bases: StrEnum

An enumeration of metrics that can be used to evaluate the performance of a retrieval system.

Attributes:

Name	Type	Description
MAP		Mean Average Precision.
NDCG		Normalized Discounted Cumulative Gain.
PRECISION		Precision.
RECALL		Recall.

all() classmethod

Returns all the metrics in the enum.

Returns:

Type	Description
list[str]	list[str]: A list of all the metrics in the enum.

trec_format(metric, k) classmethod

Returns the TREC format of the metric.

Parameters:

Name	Type	Description	Default
metric	str	The metric to convert.	required
k	int | list[int]	The number of retrieved chunks to consider.	required

Returns:

Name	Type	Description
str	str	The TREC format of the metric.

PyTrecMetric(metrics=None, k=20)

Bases: BaseMetric

Pytrec_eval metric.

Required fields: - retrieved_chunks: The retrieved chunk ids with their similarity score. - ground_truth_chunk_ids: The ground truth chunk ids.

Example:

data = RetrievalData(
    retrieved_chunks={
        "chunk1": 0.9,
        "chunk2": 0.8,
        "chunk3": 0.7,
    },
    ground_truth_chunk_ids=["chunk1", "chunk2", "chunk3"],
)
metric = PyTrecMetric()
await metric.evaluate(data)

Attributes:

Name	Type	Description
name	str	The name of the metric.
metrics	list[PyTrecEvalMetric | str] | set[PyTrecEvalMetric | str] | None	The metrics to evaluate.
k_values	int | list[int]	The number of retrieved chunks to consider.

Initializes the PyTrecMetric.

Parameters:

Name	Type	Description	Default
metrics	list[PyTrecEvalMetric | str] | set[PyTrecEvalMetric | str] | None	The metrics to evaluate. Defaults to all metrics.	None
k	int | list[int]	The number of retrieved chunks to consider. Defaults to 20.	20

convert_to_metric_name(metric)

Converts a pytrec_eval metric string into a user-friendly format.

For example, 'map_cut_20' becomes 'map@20' and 'P_20' becomes 'precision@20'.

Parameters:

Name	Type	Description	Default
metric	str	The metric string from pytrec_eval.	required

Returns:

Name	Type	Description
str	str	A user-friendly metric name.