Chunk metadata merger

Defines a helper class to merge metadata from multiple chunks.

ChunkMetadataMerger(merger_func_map=None, default_merger_func=None, retained_keys=None)

A helper class to merge metadata from multiple chunks.

Attributes:

Name	Type	Description
merger_func_map	dict[str, Callable[[list[Any]], Any]]	A mapping of metadata keys to merger functions.
default_merger_func	Callable[[list[Any]], Any]	The default merger function for metadata keys that are not present in the merger_func_map.
retained_keys	set[str] | None	The keys that should be retained in the merged metadata. If None, all intersection keys are retained.

Initializes a new instance of the ChunkMetadataMerger class.

Parameters:

Name	Type	Description	Default
merger_func_map	dict[str, Callable[[list[Any]], Any]] | None	A mapping of metadata keys to merger functions. Defaults to None, in which case a default merger map is used. The default merger map: 1. Picks the first value of the PREV_CHUNK_ID key. 2. Picks the last value of the NEXT_CHUNK_ID key.	None
default_merger_func	Callable[[list[Any]], Any] | None	The default merger for metadata keys that are not present in the merger_func_map. Defaults to None, in which case a default merger that picks the first value is used.	None
retained_keys	set[str] | None	The keys that should be retained in the merged metadata. Defaults to None, in which case all intersection keys are retained.	None

merge(metadatas)

Merges metadata from multiple chunks.

Parameters:

Name	Type	Description	Default
metadatas	list[dict[str, Any]]	The metadata to merge.	required

Returns:

Type	Description
dict[str, Any]	dict[str, Any]: The merged metadata.