Pytrec metric
Pytrec_eval metric.
PyTrecEvalMetric
Bases: StrEnum
An enumeration of metrics that can be used to evaluate the performance of a retrieval system.
Attributes:
| Name | Type | Description |
|---|---|---|
MAP |
Mean Average Precision. |
|
NDCG |
Normalized Discounted Cumulative Gain. |
|
PRECISION |
Precision. |
|
RECALL |
Recall. |
all()
classmethod
Returns all the metrics in the enum.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of all the metrics in the enum. |
trec_format(metric, k)
classmethod
Returns the TREC format of the metric.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric
|
str
|
The metric to convert. |
required |
k
|
int | list[int]
|
The number of retrieved chunks to consider. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The TREC format of the metric. |
PyTrecMetric(metrics=None, k=20)
Bases: BaseMetric
PyTrec Metric.
A wrapper for pytrec_eval to evaluate common Information Retrieval (IR) metrics.
This metric allows you to compute various standard IR scores like NDCG, MAP, MRR, Reciprocal Rank, etc., based on
retrieved chunks and ground truth chunk IDs.
Available Fields
- retrieved_chunks (dict[str, float]): The retrieved chunk ids with their similarity score.
- ground_truth_chunk_ids (list[str]): The ground truth chunk ids.
Scoring
- 0.0-1.0 (Continuous): A higher score indicates better retrieval performance.
Cookbook Example
Please refer to example_pytrec_metric.py in the gen-ai-sdk-cookbook repository.
Initializes the PyTrecMetric.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metrics
|
list[PyTrecEvalMetric | str] | set[PyTrecEvalMetric | str] | None
|
The metrics to evaluate. Defaults to all metrics. |
None
|
k
|
int | list[int]
|
The number of retrieved chunks to consider. Defaults to 20. |
20
|
convert_to_metric_name(metric)
Converts a pytrec_eval metric string into a user-friendly format.
For example, 'map_cut_20' becomes 'map@20' and 'P_20' becomes 'precision@20'.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric
|
str
|
The metric string from pytrec_eval. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
A user-friendly metric name. |