Base runner

Base class for runner.

Authors

Christina Alexandra (christina.alexandra@gdplabs.id)

References

NONE

BaseRunner(data, inference_fn, evaluators, experiment_tracker=None, batch_size=10, **kwargs)

Bases: ABC

Abstract class for runner.

This class defines the interface for all runner.

Attributes:

Name	Type	Description
data	str | BaseDataset	The data to evaluate.
inference_fn	Callable	The inference function to use.
evaluators	list[BaseEvaluator]	The evaluators to use.
experiment_tracker	BaseExperimentTracker | None	The experiment tracker.

Initialize the runner.

Parameters:

Name	Type	Description	Default
data	str | BaseDataset	The data to evaluate.	required
inference_fn	Callable	The inference function to use.	required
evaluators	list[BaseEvaluator]	The evaluators to use.	required
experiment_tracker	BaseExperimentTracker | None	The experiment tracker.	None
**kwargs	Any	Additional configuration parameters.	{}

evaluate(dataset) async

Run the evaluator on the dataset.

The dataset is evaluated in batches of the given batch size.

Parameters:

Name	Type	Description	Default
dataset	BaseDataset	The dataset to run the evaluator on.	required

Returns:

Name	Type	Description
BaseDataset	BaseDataset	The dataset with the evaluator results.