Request Processor

Modules concerning the request processor modules to perform model inferences in Gen AI applications.

LMRequestProcessor(prompt_builder, lm_invoker, output_parser=None)

A request processor to perform language models inference.

The LMRequestProcessor class handles the process of building a prompt, invoking a language model, and optionally parsing the output. It combines a prompt builder, language model invoker, and an optional output parser to manage the inference process in Gen AI applications.

Attributes:

Name	Type	Description
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.
output_parser	BaseOutputParser | None	The optional parser to process the model's output, if any.
tool_dict	dict[str, Tool]	A dictionary of tools provided to the language model to enable tool calling, if any. The dictionary maps the tool name to the tools themselves.

Initializes a new instance of the LMRequestProcessor class.

Parameters:

Name	Type	Description	Default
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.	required
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.	required
output_parser	BaseOutputParser	An optional parser to process the model's output. Defaults to None.	None

clear_response_schema()

Clears the response schema for the LM invoker.

This method clears the response schema for the LM invoker.

clear_tools()

Clears the tools for the LM invoker.

This method clears the tools for the LM invoker.

process(prompt_kwargs=None, history=None, extra_contents=None, hyperparameters=None, event_emitter=None, auto_execute_tools=True, max_lm_calls=5, **kwargs) async

Processes a language model inference request.

This method processes the language model inference request as follows: 1. Assembling the prompt using the provided keyword arguments. 2. Invoking the language model with the assembled prompt and optional hyperparameters. 3. If auto_execute_tools is True, the method will automatically execute tools if the LM output includes tool calls. 4. Optionally parsing the model's output using the output parser if provided. If the model output is an LMOutput object, the output parser will process the response attribute of the LMOutput object.

Parameters:

Name	Type	Description	Default
prompt_kwargs	dict[str, Any]	Deprecated parameter for passing prompt kwargs. Replaced by **kwargs. Defaults to None	None
history	list[Message] | None	A list of conversation history to be included in the prompt. Defaults to None.	None
extra_contents	list[MessageContent] | None	A list of extra contents to be included in the user message. Defaults to None.	None
hyperparameters	dict[str, Any] | None	A dictionary of hyperparameters for the model invocation. Defaults to None.	None
event_emitter	EventEmitter | None	An event emitter for streaming model outputs. Defaults to None.	None
auto_execute_tools	bool	Whether to automatically execute tools if the LM invokes output tool calls. Defaults to True.	True
max_lm_calls	int	The maximum number of times the language model can be invoked when auto_execute_tools is True. Defaults to 5.	5
**kwargs	Any	Keyword arguments that will be passed to format the prompt builder. Values must be either a string or an object that can be serialized to a string. Reserved keyword arguments that cannot be passed to the prompt builder include: 1. history 2. extra_contents 3. hyperparameters 4. event_emitter 5. auto_execute_tools 6. max_lm_calls	{}

Returns:

Name	Type	Description
Any	Any	The result of the language model invocation, optionally parsed by the output parser.

set_response_schema(response_schema)

Sets the response schema for the LM invoker.

This method sets the response schema for the LM invoker. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

set_tools(tools)

Sets the tools for the LM invoker.

This method sets the tools for the LM invoker. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool]	The list of tools to be used.	required

UsesLM

A mixin to be extended by components that use LMRequestProcessor.

This mixin should be extended by components that use LMRequestProcessor. Components that extend this mixin must have a constructor that accepts the LMRequestProcessor instance as its first argument.

LM based components can be categorized into two types: 1. Components that do not utilize structured output. 2. Components that utilize structured output.

Building a component without structured output

As defined above, the component must accepts an LMRequestProcessor instance as its first argument, e.g.:

class LMBasedComponent(Component, UsesLM):
    def __init__(self, lm_request_processor: LMRequestProcessor, custom_kwarg: str):
        self.lm_request_processor = lm_request_processor
        self.custom_kwarg = custom_kwarg

Using the from_lm_components method provided by this mixin, the component can be instantiated as follows:

component = LMBasedComponent.from_lm_components(
    prompt_builder,
    lm_invoker,
    output_parser,
    custom_kwarg="custom_value",
)

Building a component with structured output

When the component utilizes structured output, the _parse_structured_output method can be used to simplify the process of extracting the structured output in the component's runtime methods, e.g.:

class LMBasedComponent(Component, UsesLM):
    def __init__(self, lm_request_processor: LMRequestProcessor, custom_kwarg: str):
        self.lm_request_processor = lm_request_processor
        self.custom_kwarg = custom_kwarg

    def runtime_method(self, param1: str, param2: str) -> str:
        lm_output = self.lm_request_processor.process(param1=param1, param2=param2)
        return self._parse_structured_output(lm_output, "target_key", "fallback_output")

Notice that in the above example, the LMRequestProcessor is configured to take param1 and param2 as keyword arguments and output a structured output that contains the target_key key. Hence, these conditions must be fulfilled when instantiating the component.

This mixin also provides the with_structured_output method to simplify the process of instantiating the component with structured output. Let's take a look at an example that meets the above conditions:

class Schema(BaseModel):
    target_key: str

component = LMBasedComponent.with_structured_output(
    model_id="openai/gpt-4.1-mini",
    response_schema=Schema,
    system_template="system_template {param1}",
    user_template="user_template {param2}",
    custom_kwarg="custom_value",
)

Building a structured output preset

If desired, the component can also define a quick preset. This can be done by providing default prompts as response schema. Here's an example:

class Schema(BaseModel):
    target_key: str

@classmethod
def from_preset(cls, model_id: str, custom_kwarg: str) -> "LMBasedComponent":
    return cls.with_structured_output(
        model_id=model_id,
        response_schema=Schema,
        system_template=PRESET_SYSTEM_TEMPLATE,
        user_template=PRESET_USER_TEMPLATE,
        custom_kwarg=custom_kwarg,
    )
)

Then, the preset can be instantiated as follows:

component = LMBasedComponent.from_preset(
    model_id="openai/gpt-4.1-mini",
    custom_kwarg="custom_value",
)

from_lm_components(prompt_builder, lm_invoker, output_parser=None, **kwargs) classmethod

Creates an instance from LMRequestProcessor components directly.

This method is a shortcut to initialize the class by providing the LMRequestProcessor components directly.

Parameters:

Name	Type	Description	Default
prompt_builder	PromptBuilder	The prompt builder used to format the prompt.	required
lm_invoker	BaseLMInvoker	The language model invoker that handles the model inference.	required
output_parser	BaseOutputParser	An optional parser to process the model's output. Defaults to None.	None
**kwargs	Any	Additional keyword arguments to be passed to the class constructor.	{}

Returns:

Name	Type	Description
UsesLM	UsesLM	An instance of the class that mixes in this mixin.

with_structured_output(model_id, response_schema, system_template='', user_template='', **kwargs) classmethod

Creates an instance with structured output configuration.

This method is a shortcut to initialize the class with structured output configuration.

Parameters:

Name	Type	Description	Default
model_id	str	The model ID of the language model.	required
response_schema	type[BaseModel]	The response schema of the language model.	required
system_template	str	The system template of the language model. Defaults to an empty string.	''
user_template	str	The user template of the language model. Defaults to an empty string.	''
**kwargs	Any	Additional keyword arguments to be passed to the class constructor.	{}

Returns:

Name	Type	Description
UsesLM	UsesLM	An instance of the class that mixes in this mixin with structured output configuration.