Lm based reference formatter

Defines a reference formatter that utilizes a language model to filter the candidate chunks.

Authors

Henry Wicaksono (henry.wicaksono@gdplabs.id)

References

NONE

LMBasedReferenceFormatter(lm_request_processor, batch_size=DEFAULT_BATCH_SIZE, reference_metadata=DEFAULT_REFERENCE_METADATA, stringify=True, format_references_func=None, format_chunk_func=None, format_chunk_metadata_map=None, streamable=True, simplify_events=False)

Bases: BaseReferenceFormatter, UsesLM

A reference formatter that utilizes a language model to filter the candidate chunks.

This class uses a language model to filter the candidate chunks based on their relevance to the generated response. It then formats the relevant chunks into a reference string or a list of chunks depending on the stringify attribute.

Attributes:

Name	Type	Description
lm_request_processor	LMRequestProcessor	The request processor used to handle the candidate chunks filtering.
batch_size	int	The number of chunks to process in each batch.
stringify	bool	Whether to format the references as a string. If False, the references will be returned as a list of chunks.
format_references_func	Callable[[list[str]], str]	A function that formats a list of formatted chunks as a string of references.
format_chunk_func	Callable[[Chunk], str]	A function that formats a chunk as a reference.
streamable	bool	A flag to indicate whether the formatted references will be streamed if an event emitter is provided.

LM request processor configuration

The LM request processor for the LMBasedReferenceFormatter must take the following variables as its input: 1. response: The synthesized response. 2. context: The formatted context string of the candidate chunks in the following format: reference_metadata: content:

The reference_metadata is the metadata of the chunk that will be referenced by the language model in the filtering process, and can be customized via the reference_metadata parameter. By default, it is set to file_name.

The LM request processor must also output a list of reference metadata of the chunks that are deemed as relevant to the synthesized response, for example: python RelevantChunks( reference_metadata=["<reference_metadata_1>", "<reference_metadata_2>"] )

Using preset

This LMBasedReferenceFormatter has a preset with a predefined system template and response schema. This preset can be used for simple usage as only the model id is required. The preset utilizes the LM invoker's structured output feature and the default reference metadata.

Usage example: python reference_formatter = LMBasedReferenceFormatter.from_preset(model_id="openai/gpt-4.1-mini") reference = await reference_formatter.format(response=response, chunks=chunks)

Using LM invoker's structured output

Using the LM invoker's structured output feature is the recommended way to use the LMBasedReferenceFormatter with customized LM request processor.

Usage example: python class RelevantChunks(BaseModel): file_name: list[str] lm_request_processor = build_lm_request_processor( model_id="...", system_template="...", config={"response_schema": RelevantChunks} ) reference_formatter = LMBasedReferenceFormatter(lm_request_processor) reference = await reference_formatter.format(response=response, chunks=chunks)

Using output parser

An alternative way to use the LMBasedReferenceFormatter is to use a separate output parser. This is useful when the language model does not support native structured output feature.

Usage example: python lm_request_processor = build_lm_request_processor( model_id="...", system_template="...", output_parser=JSONOutputParser() ) reference_formatter = LMBasedReferenceFormatter(lm_request_processor) reference = await reference_formatter.format(response=response, chunks=chunks)

Initializes a new instance of the LMBasedReferenceFormatter class.

Parameters:

Name	Type	Description	Default
lm_request_processor	LMRequestProcessor	The request processor used to handle the candidate chunks filtering.	required
batch_size	int	The number of chunks to process in each batch. Defaults to DEFAULT_BATCH_SIZE.	DEFAULT_BATCH_SIZE
reference_metadata	str	The metadata of the chunk to be referenced by the language model in the filtering process. Defaults to DEFAULT_REFERENCE_METADATA.	DEFAULT_REFERENCE_METADATA
stringify	bool	Whether to format the references as a string. If False, the references will be returned as a list of chunks. Defaults to True.	True
format_references_func	Callable[[list[str]], str] | None	A function that formats a list of formatted chunks as a string of references. Defaults to None, in which case format_references_as_string will be used.	None
format_chunk_func	Callable[[Chunk], str] | None	A function that formats a chunk as a reference. Defaults to None, in which case format_chunk_as_string will be used.	None
format_chunk_metadata_map	dict[str, str] | None	A dictionary mapping the metadata keys needed by the format_chunk_func to the actual chunk metadata keys. The keys in the dictionary must match the metadata keys needed by the format_chunk_func. Defaults to None, in which case DEFAULT_FORMAT_CHUNK_METADATA_MAP will be used.	None
streamable	bool	A flag to indicate whether the formatted references will be streamed if an event emitter is provided. Defaults to True.	True
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

from_preset(model_id, **kwargs) classmethod

Creates a reference formatter with the preset system template.

Parameters:

Name	Type	Description	Default
model_id	str	The model id.	required
**kwargs	Any	Additional keyword arguments.	{}

Returns:

Name	Type	Description
LMBasedReferenceFormatter	LMBasedReferenceFormatter	A reference formatter with the preset system template.