Lm request processor

Defines a request processor to perform language models inference.

FallbackLMRequestProcessor(prompt_builder, lm_invoker, output_parser=None, fallback_lmrp=None)

Bases: LMRequestProcessor

An LMRequestProcessor that cascades through fallback processors on invoker errors.

Wraps a primary LMRequestProcessor and an ordered list of fallback processors. On BaseInvokerError or TimeoutError from the primary, each fallback is tried in order. The last exception is re-raised if all processors fail.

Attributes:

Name	Type	Description
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.
output_parser	BaseOutputParser | None	The optional parser to process the model's output, if any.
tool_dict	dict[str, Tool]	A dictionary of tools provided to the language model to enable tool calling, if any. The dictionary maps the tool name to the tools themselves.

Initialize the fallback processor.

Parameters:

Name	Type	Description	Default
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.	required
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.	required
output_parser	BaseOutputParser | None	The optional parser to process the model's output, if any. Defaults to None.	None
fallback_lmrp	list[LMRequestProcessor] | None	Ordered list of fallback processors. Defaults to None.	None

process(*args, **kwargs) async

Process the request with fallback logic.

Tries the primary processor first, then falls back to each fallback in order on BaseInvokerError or TimeoutError. Re-raises the last exception if all fail.

Parameters:

Name	Type	Description	Default
*args	Any	Positional arguments to pass to the processor.	()
**kwargs	Any	Keyword arguments to pass to the processor.	{}

Returns:

Name	Type	Description
Any	Any	The result from the first successful processor.

Raises:

Type	Description
Exception	The last exception encountered if all processors fail.

LMRequestProcessor(prompt_builder, lm_invoker, output_parser=None)

A request processor to perform language models inference.

The LMRequestProcessor class handles the process of building a prompt, invoking a language model, and optionally parsing the output. It combines a prompt builder, language model invoker, and an optional output parser to manage the inference process in Gen AI applications.

Attributes:

Name	Type	Description
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.
output_parser	BaseOutputParser | None	The optional parser to process the model's output, if any.
tool_dict	dict[str, Tool]	A dictionary of tools provided to the language model to enable tool calling, if any. The dictionary maps the tool name to the tools themselves.

Initializes a new instance of the LMRequestProcessor class.

Parameters:

Name	Type	Description	Default
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.	required
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.	required
output_parser	BaseOutputParser	An optional parser to process the model's output. Defaults to None.	None

clear_response_schema()

Clears the response schema for the LM invoker.

This method clears the response schema for the LM invoker.

clear_tools()

Clears the tools for the LM invoker.

This method clears the tools for the LM invoker.

process(prompt_kwargs=None, history=None, extra_contents=None, hyperparameters=None, event_emitter=None, auto_execute_tools=True, max_lm_calls=5, **kwargs) async

Processes a language model inference request.

This method processes the language model inference request as follows: 1. Assembling the prompt using the provided keyword arguments. 2. Invoking the language model with the assembled prompt and optional hyperparameters. 3. If auto_execute_tools is True, the method will automatically execute tools if the LM output includes tool calls. 4. Optionally parsing the model's output using the output parser if provided. If the model output is an LMOutput object, the output parser will process the text attribute of the LMOutput object.

Parameters:

Name	Type	Description	Default
prompt_kwargs	dict[str, Any]	Deprecated parameter for passing prompt kwargs. Replaced by **kwargs. Defaults to None	None
history	list[Message] | None	A list of conversation history to be included in the prompt. Defaults to None.	None
extra_contents	list[MessageContent] | None	A list of extra contents to be included in the user message. Defaults to None.	None
hyperparameters	dict[str, Any] | None	A dictionary of hyperparameters for the model invocation. Defaults to None.	None
event_emitter	EventEmitter | None	An event emitter for streaming model outputs. Defaults to None.	None
auto_execute_tools	bool	Whether to automatically execute tools if the LM invokes output tool calls. Defaults to True.	True
max_lm_calls	int	The maximum number of times the language model can be invoked when auto_execute_tools is True. Defaults to 5.	5
**kwargs	Any	Keyword arguments that will be passed to format the prompt builder. Values must be either a string or an object that can be serialized to a string. Reserved keyword arguments that cannot be passed to the prompt builder include: 1. history 2. extra_contents 3. hyperparameters 4. event_emitter 5. auto_execute_tools 6. max_lm_calls	{}

Returns:

Name	Type	Description
Any	Any	The result of the language model invocation, optionally parsed by the output parser.

set_response_schema(response_schema)

Sets the response schema for the LM invoker.

This method sets the response schema for the LM invoker. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

set_tools(tools)

Sets the tools for the LM invoker.

This method sets the tools for the LM invoker. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool]	The list of tools to be used.	required