Lm Invoker

Modules concerning the language model invokers used in Gen AI applications.

`AnthropicLMInvoker(model_name, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, thinking=False, thinking_budget=DEFAULT_THINKING_BUDGET, use_thinking=False, output_thinking=False, bind_tools_params=None, with_structured_output_params=None)`

Bases: BaseLMInvoker

A language model invoker to interact with Anthropic language models.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`AsyncAnthropic`	The Anthropic client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Tool]`	Tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.
`thinking`	`bool`	Whether to enable thinking. Only allowed for thinking models.
`thinking_budget`	`int`	The tokens allocated for the thinking process. Only allowed for thinking models.
`output_thinking`	`bool`	Whether to output the thinking token. Will be removed in v0.5.0, where the thinking token will always be included in the output when thinking is enabled.

Basic usage

The AnthropicLMInvoker can be used as follows:

lm_invoker = AnthropicLMInvoker(model_name="claude-sonnet-4-20250514")
result = await lm_invoker.invoke("Hi there!")

Input types

The AnthropicLMInvoker supports the following input types: 1. Text. 2. Document: ".pdf". 3. Image: ".jpg", ".jpeg", ".png", ".gif", and ".webp". Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = AnthropicLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

Structured output is achieved by providing the schema name in the tool_choice parameter. This forces the model to call the provided schema as a tool. Thus, structured output is not compatible with: 1. Tool calling, since the tool calling is reserved to force the model to call the provided schema as a tool. 2. Thinking, since thinking is not allowed when a tool use is forced through the tool_choice parameter. The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = AnthropicLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = AnthropicLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The AnthropicLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = AnthropicLMInvoker(..., retry_config=retry_config)

Thinking

Thinking is a feature that allows the language model to have enhanced reasoning capabilities for complex tasks, while also providing transparency into its step-by-step thought process before it delivers its final answer. This feature is only available for certain models, starting from Claude 3.7 Sonnet. It can be enabled by setting the thinking parameter to True.

When thinking is enabled, the amount of tokens allocated for the thinking process can be set via the thinking_budget parameter. The thinking_budget: 1. Must be greater than or equal to 1024. 2. Must be less than the max_tokens hyperparameter, as the thinking_budget is allocated from the max_tokens. For example, if max_tokens=2048 and thinking_budget=1024, the language model will allocate at most 1024 tokens for thinking and the remaining 1024 tokens for generating the response.

When enabled, the reasoning is stored in the reasoning attribute in the output. Note: Before v0.5.0, this has to be set explicitly via the output_thinking parameter. Starting from v0.5.0, the reasoning will always be included in the output when thinking is enabled.

Usage example:

lm_invoker = AnthropicLMInvoker(..., thinking=True, thinking_budget=1024)

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(type="thinking", reasoning="Let me think about it...", signature="x")],
)

When streaming is enabled, the thinking token will be streamed with the EventType.DATA event type.

Streaming output example:

{"type": "data", "value": '{"data_type": "thinking_start", "data_value": ""}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "Let me think "}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "about it..."}', ...}
{"type": "data", "value": '{"data_type": "thinking_end", "data_value": ""}', ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Output types

The output of the AnthropicLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage information, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any]): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to an empty dictionary. 2.7. reasoning (list[Reasoning]): The reasoning objects, if the thinking parameter is set to True. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes the AnthropicLmInvoker instance.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the Anthropic language model.	required
`api_key`	`str \| None`	The Anthropic API key. Defaults to None, in which case the `ANTHROPIC_API_KEY` environment variable will be used.	`None`
`model_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments for the Anthropic client.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`thinking`	`bool`	Whether to enable thinking. Only allowed for thinking models. Defaults to False.	`False`
`thinking_budget`	`int`	The tokens allocated for the thinking process. Must be greater than or equal to 1024. Only allowed for thinking models. Defaults to DEFAULT_THINKING_BUDGET.	`DEFAULT_THINKING_BUDGET`
`use_thinking`	`bool`	Deprecated parameter to enable thinking. Defaults to False.	`False`
`output_thinking`	`bool`	Deprecated parameter to output the thinking token. Starting from v0.5.0, the thinking token will always be included in the output when thinking is enabled. Defaults to False.	`False`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`
2. `thinking` is True and `tools` are provided, but `output_thinking` is False. # TODO	Remove in v0.5.0

`set_response_schema(response_schema)`

Sets the response schema for the Anthropic language model.

This method sets the response schema for the Anthropic language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

Raises:

Type	Description
`ValueError`	If `tools` exists.

`set_tools(tools)`

Sets the tools for the Anthropic language model.

This method sets the tools for the Anthropic language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
`tools`	`list[Tool]`	The list of tools to be used.	required

Raises:

Type	Description
`ValueError`	If `response_schema` exists.

`AzureOpenAILMInvoker(azure_endpoint, azure_deployment, api_key=None, api_version=DEFAULT_AZURE_OPENAI_API_VERSION, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, reasoning_summary=None, bind_tools_params=None, with_structured_output_params=None)`

Bases: OpenAILMInvoker

A language model invoker to interact with Azure OpenAI language models.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the Azure OpenAI language model deployment.
`client`	`AsyncAzureOpenAI`	The Azure OpenAI client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort.
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated.
`code_interpreter`	`bool`	Whether to enable the code interpreter. Currently not supported.
`web_search`	`bool`	Whether to enable the web search. Currently not supported.

Basic usage

The AzureOpenAILMInvoker can be used as follows:

lm_invoker = AzureOpenAILMInvoker(
    azure_endpoint="https://<your-azure-openai-endpoint>.openai.azure.com/",
    azure_deployment="<your-azure-openai-deployment>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

Text.
Document: ".pdf".
Image: ".jpg", ".jpeg", ".png", ".gif", and ".webp". Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = AzureOpenAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = AzureOpenAILMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = AzureOpenAILMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"status": "completed", "incomplete_details": {"reason": None}},
)

Retry and timeout

The AzureOpenAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = AzureOpenAILMInvoker(..., retry_config=retry_config)

Reasoning

Azure OpenAI's o-series models are classified as reasoning models. Reasoning models think before they answer, producing a long internal chain of thought before responding to the user. Reasoning models excel in complex problem solving, coding, scientific reasoning, and multi-step planning for agentic workflows.

The reasoning effort of reasoning models can be set via the reasoning_effort parameter. This parameter will guide the models on how many reasoning tokens it should generate before creating a response to the prompt. Available options include: 1. "low": Favors speed and economical token usage. 2. "medium": Favors a balance between speed and reasoning accuracy. 3. "high": Favors more complete reasoning at the cost of more tokens generated and slower responses. When not set, the reasoning effort will be equivalent to medium by default.

Azure OpenAI doesn't expose the raw reasoning tokens. However, the summary of the reasoning tokens can still be generated. The summary level can be set via the reasoning_summary parameter. Available options include: 1. "auto": The model decides the summary level automatically. 2. "detailed": The model will generate a detailed summary of the reasoning tokens. Reasoning summary is not compatible with tool calling. When enabled, the reasoning summary will be stored in the reasoning attribute in the output.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(id="x", reasoning="Let me think about it...")],
)

When streaming is enabled along with reasoning summary, the reasoning summary token will be streamed with the EventType.DATA event type.

Streaming output example:

{"type": "data", "value": "Let me think ", ...}  # Reasoning summary token
{"type": "data", "value": "about it...", ...}  # Reasoning summary token
{"type": "response", "value": "Golden retriever ", ...}  # Response token
{"type": "response", "value": "is a good dog breed.", ...}  # Response token

Setting reasoning-related parameters for non-reasoning models will raise an error.

Output types

The output of the AzureOpenAILMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects, if the reasoning_summary parameter is provided for reasoning models. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the AzureOpenAILMInvoker class.

Parameters:

Name	Type	Description	Default
`azure_endpoint`	`str`	The endpoint of the Azure OpenAI service.	required
`azure_deployment`	`str`	The deployment name of the Azure OpenAI service.	required
`api_key`	`str \| None`	The API key for authenticating with Azure OpenAI. Defaults to None, in which case the `AZURE_OPENAI_API_KEY` environment variable will be used.	`None`
`api_version`	`str`	The API version of the Azure OpenAI service. Defaults to `DEFAULT_AZURE_OPENAI_API_VERSION`.	`DEFAULT_AZURE_OPENAI_API_VERSION`
`model_kwargs`	`dict[str, Any] \| None`	Additional model parameters. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	`None`
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated. Defaults to None.	`None`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`

`BedrockLMInvoker(model_name, access_key_id=None, secret_access_key=None, region_name='us-east-1', model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None)`

Bases: BaseLMInvoker

A language model invoker to interact with AWS Bedrock language models.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`session`	`Session`	The Bedrock client session.
`client_kwargs`	`dict[str, Any]`	The Bedrock client kwargs.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Tool]`	Tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.

Basic usage

The BedrockLMInvoker can be used as follows:

lm_invoker = BedrockLMInvoker(
    model_name="us.anthropic.claude-sonnet-4-20250514-v1:0",
    aws_access_key_id="<your-aws-access-key-id>",
    aws_secret_access_key="<your-aws-secret-access-key>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The BedrockLMInvoker supports the following input types: 1. Text. 2. Document: ".pdf", ".csv", ".doc", ".docx", ".xls", ".xlsx", ".html", ".txt", ".md". 3. Image: ".png", ".jpeg", ".gif", ".webp". 4. Video: ".mkv", ".mov", ".mp4", ".webm", ".flv", ".mpeg", ".mpg", ".wmv", ".three_gp". Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = BedrockLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

Structured output is achieved by providing the schema name in the tool_choice parameter. This forces the model to call the provided schema as a tool. Thus, structured output is not compatible with tool calling, since the tool calling is reserved to force the model to call the provided schema as a tool. The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = BedrockLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = BedrockLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The BedrockLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = BedrockLMInvoker(..., retry_config=retry_config)

Output types

The output of the BedrockLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage information, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any]): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to an empty dictionary. 2.7. reasoning (list[Reasoning]): The reasoning objects. Currently not supported. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes the BedrockLMInvoker instance.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the Bedrock language model.	required
`access_key_id`	`str \| None`	The AWS access key ID. Defaults to None, in which case the `AWS_ACCESS_KEY_ID` environment variable will be used.	`None`
`secret_access_key`	`str \| None`	The AWS secret access key. Defaults to None, in which case the `AWS_SECRET_ACCESS_KEY` environment variable will be used.	`None`
`region_name`	`str`	The AWS region name. Defaults to "us-east-1".	`'us-east-1'`
`model_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments for the Bedrock client.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`

Raises:

Type	Description
`ValueError`	If `response_schema` is provided, but `tools` are also provided.
`ValueError`	If `access_key_id` or `secret_access_key` is neither provided nor set in the `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY` environment variables, respectively.

`set_response_schema(response_schema)`

Sets the response schema for the Bedrock language model.

This method sets the response schema for the Bedrock language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

Raises:

Type	Description
`ValueError`	If `tools` exists.

`set_tools(tools)`

Sets the tools for the Bedrock language model.

This method sets the tools for the Bedrock language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
`tools`	`list[Tool]`	The list of tools to be used.	required

Raises:

Type	Description
`ValueError`	If `response_schema` exists.

`DatasaurLMInvoker(base_url, api_key=None, model_kwargs=None, default_hyperparameters=None, output_analytics=False, retry_config=None, citations=False)`

Bases: OpenAICompatibleLMInvoker

A language model invoker to interact with Datasaur LLM Projects Deployment API.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`AsyncOpenAI`	The OpenAI client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling. Currently not supported.
`response_schema`	`ResponseSchema \| None`	The schema of the response. Currently not supported.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.
`citations`	`bool`	Whether to output the citations.

Basic usage

The DatasaurLMInvoker can be used as follows:

lm_invoker = DatasaurLMInvoker(base_url="https://deployment.datasaur.ai/api/deployment/teamId/deploymentId/")
result = await lm_invoker.invoke("Hi there!")

Input types

Text.
Audio, with extensions depending on the language model's capabilities.
Image, with extensions depending on the language model's capabilities.
Document, with extensions depending on the language model's capabilities. Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"finish_reason": "stop"},
)

When streaming is enabled, token usage is not supported. Therefore, the token_usage attribute will be None regardless of the value of the output_analytics parameter.

Retry and timeout

The DatasaurLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = DatasaurLMInvoker(..., retry_config=retry_config)

Citations

The DatasaurLMInvoker can be configured to output the citations used to generate the response. They can be enabled by setting the citations parameter to True. When enabled, the citations will be stored as Chunk objects in the citations attribute in the output.

Usage example:

lm_invoker = DatasaurLMInvoker(..., citations=True)

Output example:

LMOutput(
    response="The winner of the match is team A ([Example title](https://www.example.com)).",
    citations=[Chunk(id="123", content="...", metadata={...}, score=0.95)],
)

Output types

The output of the DatasaurLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls. Currently not supported. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output. Currently not supported. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects. Currently not supported. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the DatasaurLMInvoker class.

Parameters:

Name	Type	Description	Default
`base_url`	`str`	The base URL of the Datasaur LLM Projects Deployment API.	required
`api_key`	`str \| None`	The API key for authenticating with Datasaur LLM Projects Deployment API. Defaults to None, in which case the `DATASAUR_API_KEY` environment variable will be used.	`None`
`model_kwargs`	`dict[str, Any] \| None`	Additional model parameters. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`citations`	`bool`	Whether to output the citations. Defaults to False.	`False`

Raises:

Type	Description
`ValueError`	If the `api_key` is not provided and the `DATASAUR_API_KEY` environment variable is not set.

`set_response_schema(response_schema)`

Sets the response schema for the Datasaur LLM Projects Deployment API.

This method is raises a NotImplementedError because the Datasaur LLM Projects Deployment API does not support response schema.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

Raises:

Type	Description
`NotImplementedError`	This method is not supported for the Datasaur LLM Projects Deployment API.

`set_tools(tools)`

Sets the tools for the Datasaur LLM Projects Deployment API.

This method is raises a NotImplementedError because the Datasaur LLM Projects Deployment API does not support tools.

Parameters:

Name	Type	Description	Default
`tools`	`list[Tool]`	The list of tools to be used.	required

Raises:

Type	Description
`NotImplementedError`	This method is not supported for the Datasaur LLM Projects Deployment API.

`GoogleGenerativeAILMInvoker(model_name, api_key, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, bind_tools_params=None, with_structured_output_params=None)`

Bases: GoogleLMInvoker

A language model invoker to interact with Google Gen AI language models.

This class has been deprecated as Google Generative AI is now supported through GoogleLMInvoker. This class is maintained for backward compatibility and will be removed in version 0.5.0.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`Client`	The Google client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.

Initializes a new instance of the GoogleGenerativeAILMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the multimodal language model to be used.	required
`api_key`	`str`	The API key for authenticating with Google Gen AI.	required
`model_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments for the Google Generative AI client.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

`GoogleLMInvoker(model_name, api_key=None, credentials_path=None, project_id=None, location='us-central1', model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, thinking=None, thinking_budget=DEFAULT_THINKING_BUDGET, bind_tools_params=None, with_structured_output_params=None)`

Bases: BaseLMInvoker

A language model invoker to interact with Google language models.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client_params`	`dict[str, Any]`	The Google client instance init parameters.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.
`thinking`	`bool`	Whether to enable thinking. Only allowed for thinking models.
`thinking_budget`	`int`	The tokens allowed for thinking process. Only allowed for thinking models. If set to -1, the model will control the budget automatically.

Basic usage

The GoogleLMInvoker can be used as follows:

lm_invoker = GoogleLMInvoker(model_name="gemini-2.5-flash")
result = await lm_invoker.invoke("Hi there!")

Authentication

The GoogleLMInvoker can use either Google Gen AI or Google Vertex AI.

Google Gen AI is recommended for quick prototyping and development. It requires a Gemini API key for authentication.

Usage example:

lm_invoker = GoogleLMInvoker(
    model_name="gemini-2.5-flash",
    api_key="your_api_key"
)

Google Vertex AI is recommended to build production-ready applications. It requires a service account JSON file for authentication.

Usage example:

lm_invoker = GoogleLMInvoker(
    model_name="gemini-2.5-flash",
    credentials_path="path/to/service_account.json"
)

If neither api_key nor credentials_path is provided, Google Gen AI will be used by default. The GOOGLE_API_KEY environment variable will be used for authentication.

Input types

Text.
Audio: ".aac", ".flac", ".mp3", and ".wav".
Document: ".pdf", ".txt", ".csv", ".md", ".css", ".html", and ".xml".
Image: ".jpg", ".jpeg", ".png", and ".webp".
Video: ".x-flv", ".mpeg", ".mpg", ".mp4", ".webm", ".wmv", and ".3gpp". Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can be passed with either the user or assistant role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = GoogleLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

Structured output is not compatible with tool calling. The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = GoogleLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = GoogleLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"finish_reason": "STOP", "finish_message": None},
)

Retry and timeout

The GoogleLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = GoogleLMInvoker(..., retry_config=retry_config)

Thinking

Thinking is a feature that allows the language model to have enhanced reasoning capabilities for complex tasks, while also providing transparency into its step-by-step thought process before it delivers its final answer. It can be enabled by setting the thinking parameter to True.

Thinking is only available for certain models, starting from Gemini 2.5 series, and is required for Gemini 2.5 Pro models. Therefore, thinking defaults to True for Gemini 2.5 Pro models and False for other models. Setting thinking to False for Gemini 2.5 Pro models will raise a ValueError. When enabled, the reasoning is stored in the reasoning attribute in the output.

Usage example:

lm_invoker = GoogleLMInvoker(..., thinking=True, thinking_budget=1024)

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(reasoning="Let me think about it...")],
)

When streaming is enabled, the thinking token will be streamed with the EventType.DATA event type.

Streaming output example:

{"type": "data", "value": '{"data_type": "thinking_start", "data_value": ""}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "Let me think "}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "about it..."}', ...}
{"type": "data", "value": '{"data_type": "thinking_end", "data_value": ""}', ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

When thinking is enabled, the amount of tokens allocated for the thinking process can be set via the thinking_budget parameter. The thinking_budget: 1. Defaults to -1, in which case the model will control the budget automatically. 2. Must be greater than the model's minimum thinking budget. For more details, please refer to https://ai.google.dev/gemini-api/docs/thinking

Output types

The output of the GoogleLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects, if the thinking parameter is set to True. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the GoogleLMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the model to use.	required
`api_key`	`str \| None`	Required for Google Gen AI authentication. Cannot be used together with `credentials_path`. Defaults to None.	`None`
`credentials_path`	`str \| None`	Required for Google Vertex AI authentication. Path to the service account credentials JSON file. Cannot be used together with `api_key`. Defaults to None.	`None`
`project_id`	`str \| None`	The Google Cloud project ID for Vertex AI. Only used when authenticating with `credentials_path`. Defaults to None, in which case it will be loaded from the credentials file.	`None`
`location`	`str`	The location of the Google Cloud project for Vertex AI. Only used when authenticating with `credentials_path`. Defaults to "us-central1".	`'us-central1'`
`model_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments for the Google Vertex AI client.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`thinking`	`bool \| None`	Whether to enable thinking. Only allowed for thinking models. Defaults to True for Gemini 2.5 Pro models and False for other models.	`None`
`thinking_budget`	`int`	The tokens allowed for thinking process. Only allowed for thinking models. Defaults to -1, in which case the model will control the budget automatically.	`DEFAULT_THINKING_BUDGET`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

Note

If neither api_key nor credentials_path is provided, Google Gen AI will be used by default. The GOOGLE_API_KEY environment variable will be used for authentication.

`set_response_schema(response_schema)`

Sets the response schema for the Google language model.

This method sets the response schema for the Google language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

Raises:

Type	Description
`ValueError`	If `tools` exists.

`set_tools(tools)`

Sets the tools for the Google language model.

This method sets the tools for the Google language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
`tools`	`list[Tool]`	The list of tools to be used.	required

Raises:

Type	Description
`ValueError`	If `response_schema` exists.

`GoogleVertexAILMInvoker(model_name, credentials_path, project_id=None, location='us-central1', model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, bind_tools_params=None, with_structured_output_params=None)`

Bases: GoogleLMInvoker

A language model invoker to interact with Google Vertex AI language models.

This class has been deprecated as Google Vertex AI is now supported through GoogleLMInvoker. This class is maintained for backward compatibility and will be removed in version 0.5.0.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`Client`	The Google client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.

Initializes a new instance of the GoogleVertexAILMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the multimodal language model to be used.	required
`credentials_path`	`str`	The path to the Google Cloud service account credentials JSON file.	required
`project_id`	`str \| None`	The Google Cloud project ID. Defaults to None, in which case the project ID will be loaded from the credentials file.	`None`
`location`	`str`	The location of the Google Cloud project. Defaults to "us-central1".	`'us-central1'`
`model_kwargs`	`dict[str, Any] \| None`	Additional keyword arguments for the Google Vertex AI client.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

`LangChainLMInvoker(model=None, model_class_path=None, model_name=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, llm=None, bind_tools_params=None, with_structured_output_params=None)`

Bases: BaseLMInvoker

A language model invoker to interact with LangChain's BaseChatModel.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`model`	`BaseChatModel`	The LangChain's BaseChatModel instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.

Basic usage

The LangChainLMInvoker can be used as follows:

lm_invoker = LangChainLMInvoker(
    model_class_path="langchain_openai.ChatOpenAI",
    model_name="gpt-4.1-nano",
)
result = await lm_invoker.invoke("Hi there!")

Initialization

The LangChainLMInvoker can be initialized by either passing:

A LangChain's BaseChatModel instance: Usage example:

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4.1-nano", api_key="your_api_key")
lm_invoker = LangChainLMInvoker(model=model)

A model path in the format of ".": Usage example:

lm_invoker = LangChainLMInvoker(
    model_class_path="langchain_openai.ChatOpenAI",
    model_name="gpt-4.1-nano",
    model_kwargs={"api_key": "your_api_key"}
)

For the list of supported providers, please refer to the following table: https://python.langchain.com/docs/integrations/chat/#featured-providers

Input types

Text.
Image, with extensions depending on the language model's capabilities. Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with specific roles, depending on the language model's capabilities.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = LangChainLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

Structured output is not compatible with tool calling. The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = LangChainLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = LangChainLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"finish_reason": "stop"},
)

Retry and timeout

The LangChainLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = LangChainLMInvoker(..., retry_config=retry_config)

Output types

The output of the LangChainLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects. Currently not supported. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the LangChainLMInvoker class.

Parameters:

Name	Type	Description	Default
`model`	`BaseChatModel \| None`	The LangChain's BaseChatModel instance. If provided, will take precedence over the `model_class_path` parameter. Defaults to None.	`None`
`model_class_path`	`str \| None`	The LangChain's BaseChatModel class path. Must be formatted as "." (e.g. "langchain_openai.ChatOpenAI"). Ignored if `model` is provided. Defaults to None.	`None`
`model_name`	`str \| None`	The model name. Only used if `model_class_path` is provided. Defaults to None.	`None`
`model_kwargs`	`dict[str, Any] \| None`	The additional keyword arguments. Only used if `model_class_path` is provided. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`llm`	`BaseChatModel \| None`	Deprecated parameter to pass the LangChain's BaseChatModel instance. Equivalent to the `model` parameter. Retained for backward compatibility. Defaults to None.	`None`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`	If `response_schema` is provided, but `tools` are also provided.

`set_response_schema(response_schema)`

Sets the response schema for the LangChain's BaseChatModel.

This method sets the response schema for the LangChain's BaseChatModel. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

Raises:

Type	Description
`ValueError`	If `tools` exists.

`set_tools(tools)`

Sets the tools for LangChain's BaseChatModel.

This method sets the tools for LangChain's BaseChatModel. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
`tools`	`list[Tool]`	The list of tools to be used.	required

Raises:

Type	Description
`ValueError`	If `response_schema` exists.

`LiteLLMLMInvoker(model_id, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None)`

Bases: OpenAICompatibleLMInvoker

A language model invoker to interact with language models using LiteLLM.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`completion`	`function`	The LiteLLM's completion function.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Tool]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.

Basic usage

The LiteLLMLMInvoker can be used as follows:

lm_invoker = LiteLLMLMInvoker(model_id="openai/gpt-4.1-nano")
result = await lm_invoker.invoke("Hi there!")

Initialization

The LiteLLMLMInvoker provides an interface to interact with multiple language model providers. In order to use this class: 1. The model_id parameter must be in the format of provider/model_name. e.g. openai/gpt-4o-mini. 2. The required credentials must be provided via the environment variables.

Usage example:

os.environ["OPENAI_API_KEY"] = "your_openai_api_key"
lm_invoker = LiteLLMLMInvoker(model_id="openai/gpt-4o-mini")

For the complete list of supported providers and their required credentials, please refer to the LiteLLM documentation: https://docs.litellm.ai/docs/providers/

Input types

Text.
Audio, with extensions depending on the language model's capabilities.
Image, with extensions depending on the language model's capabilities. Non-text inputs must be of valid file extensions and can be passed as a Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = LiteLLMLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = LiteLLMLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = LiteLLMLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"finish_reason": "stop"},
)

When streaming is enabled, token usage is not supported. Therefore, the token_usage attribute will be None regardless of the value of the output_analytics parameter.

Retry and timeout

The LiteLLMLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = LiteLLMLMInvoker(..., retry_config=retry_config)

Reasoning: Some language models support advanced reasoning capabilities. When using such reasoning-capable models, you can configure how much reasoning the model should perform before generating a final response by setting reasoning-related parameters.

The reasoning effort of reasoning models can be set via the `reasoning_effort` parameter. This parameter
will guide the models on how many reasoning tokens it should generate before creating a response to the prompt.
The reasoning effort is only supported by some language models.
Available options include:
1. "low": Favors speed and economical token usage.
2. "medium": Favors a balance between speed and reasoning accuracy.
3. "high": Favors more complete reasoning at the cost of more tokens generated and slower responses.
This may differ between models. When not set, the reasoning effort will be equivalent to None by default.

When using reasoning models, some providers might output the reasoning summary. These will be stored in the
`reasoning` attribute in the output.

Output example:
```python
LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(id="", reasoning="Let me think about it...")],
)
```

When streaming is enabled along with reasoning and the provider supports reasoning output, the reasoning token
will be streamed with the `EventType.DATA` event type.

Streaming output example:
```python
{"type": "data", "value": '{"data_type": "thinking_start", "data_value": ""}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "Let me think "}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "about it..."}', ...}
{"type": "data", "value": '{"data_type": "thinking_end", "data_value": ""}', ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Setting reasoning-related parameters for non-reasoning models will raise an error.

Output types: The output of the LiteLLMLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects. Currently not supported. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the LiteLLMLMInvoker class.

Parameters:

Name	Type	Description	Default
`model_id`	`str`	The ID of the model to use. Must be in the format of `provider/model_name`.	required
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Defaults to None.	`None`

`OpenAICompatibleLMInvoker(model_name, base_url, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, bind_tools_params=None, with_structured_output_params=None)`

Bases: BaseLMInvoker

A language model invoker to interact with endpoints compatible with OpenAI's chat completion API contract.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`AsyncOpenAI`	The OpenAI client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model.

When to use

The OpenAICompatibleLMInvoker is designed to interact with endpoints that are compatible with OpenAI's chat completion API contract. This includes but are not limited to: 1. DeepInfra (https://deepinfra.com/) 2. DeepSeek (https://deepseek.com/) 3. Groq (https://groq.com/) 4. OpenRouter (https://openrouter.ai/) 5. Text Generation Inference (https://github.com/huggingface/text-generation-inference) 6. Together.ai (https://together.ai/) 7. vLLM (https://vllm.ai/) When using this invoker, please note that the supported features and capabilities may vary between different endpoints and language models. Using features that are not supported by the endpoint will result in an error.

Basic usage

The OpenAICompatibleLMInvoker can be used as follows:

lm_invoker = OpenAICompatibleLMInvoker(
    model_name="llama3-8b-8192",
    base_url="https://api.groq.com/openai/v1",
    api_key="<your-api-key>"
)
result = await lm_invoker.invoke("Hi there!")

Input types

Text.
Audio, with extensions depending on the language model's capabilities.
Image, with extensions depending on the language model's capabilities. Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = OpenAICompatibleLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = OpenAICompatibleLMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = OpenAICompatibleLMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"finish_reason": "stop"},
)

When streaming is enabled, token usage is not supported. Therefore, the token_usage attribute will be None regardless of the value of the output_analytics parameter.

Retry and timeout

The OpenAICompatibleLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = OpenAICompatibleLMInvoker(..., retry_config=retry_config)

Reasoning

Some language models support advanced reasoning capabilities. When using such reasoning-capable models, you can configure how much reasoning the model should perform before generating a final response by setting reasoning-related parameters.

The reasoning effort of reasoning models can be set via the reasoning_effort parameter. This parameter will guide the models on how many reasoning tokens it should generate before creating a response to the prompt. The reasoning effort is only supported by some language models. Available options include: 1. "low": Favors speed and economical token usage. 2. "medium": Favors a balance between speed and reasoning accuracy. 3. "high": Favors more complete reasoning at the cost of more tokens generated and slower responses. This may differ between models. When not set, the reasoning effort will be equivalent to None by default.

When using reasoning models, some providers might output the reasoning summary. These will be stored in the reasoning attribute in the output.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(id="", reasoning="Let me think about it...")],
)

When streaming is enabled along with reasoning and the provider supports reasoning output, the reasoning token will be streamed with the EventType.DATA event type.

Streaming output example: ```python {"type": "data", "value": '{"data_type": "thinking_start", "data_value": ""}', ...} {"type": "data", "value": '{"data_type": "thinking", "data_value": "Let me think "}', ...} {"type": "data", "value": '{"data_type": "thinking", "data_value": "about it..."}', ...} {"type": "data", "value": '{"data_type": "thinking_end", "data_value": ""}', ...} {"type": "response", "value": "Golden retriever ", ...} {"type": "response", "value": "is a good dog breed.", ...}

Setting reasoning-related parameters for non-reasoning models will raise an error.

Output types

The output of the OpenAICompatibleLMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects. Currently not supported. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations. Currently not supported. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results. Currently not supported. Defaults to an empty list.

Initializes a new instance of the OpenAICompatibleLMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the language model hosted on the OpenAI compatible endpoint.	required
`base_url`	`str`	The base URL for the OpenAI compatible endpoint.	required
`api_key`	`str \| None`	The API key for authenticating with the OpenAI compatible endpoint. Defaults to None, in which case the `OPENAI_API_KEY` environment variable will be used.	`None`
`model_kwargs`	`dict[str, Any] \| None`	Additional model parameters. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`reasoning_effort`	`str \| None`	The reasoning effort for the language model. Defaults to None.	`None`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

`set_response_schema(response_schema)`

Sets the response schema for the language model hosted on the OpenAI compatible endpoint.

This method sets the response schema for the language model hosted on the OpenAI compatible endpoint. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

`OpenAILMInvoker(model_name, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, reasoning_summary=None, code_interpreter=False, web_search=False, bind_tools_params=None, with_structured_output_params=None)`

Bases: BaseLMInvoker

A language model invoker to interact with OpenAI language models.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`AsyncOpenAI`	The OpenAI client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort.
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated.
`code_interpreter`	`bool`	Whether to enable the code interpreter.
`web_search`	`bool`	Whether to enable the web search.

Basic usage

The OpenAILMInvoker can be used as follows:

lm_invoker = OpenAILMInvoker(model_name="gpt-4.1-nano")
result = await lm_invoker.invoke("Hi there!")

Input types

Text.
Document: ".pdf".
Image: ".jpg", ".jpeg", ".png", ".gif", and ".webp". Non-text inputs must be of valid file extensions and can be passed as an Attachment object.

Non-text inputs can only be passed with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")

prompt = [(PromptRole.USER, [text, image])]
result = await lm_invoker.invoke(prompt)

Tool calling

Tool calling is a feature that allows the language model to call tools to perform tasks. Tools can be passed to the via the tools parameter as a list of LangChain's Tool objects. When tools are provided and the model decides to call a tool, the tool calls are stored in the tool_calls attribute in the output.

Usage example:

lm_invoker = OpenAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    response="Let me call the tools...",
    tool_calls=[
        ToolCall(id="123", name="tool_1", args={"key": "value"}),
        ToolCall(id="456", name="tool_2", args={"key": "value"}),
    ]
)

Structured output

Structured output is a feature that allows the language model to output a structured response. This feature can be enabled by providing a schema to the response_schema parameter.

The schema must be either a JSON schema dictionary or a Pydantic BaseModel class. If JSON schema is used, it must be compatible with Pydantic's JSON schema, especially for complex schemas. For this reason, it is recommended to create the JSON schema using Pydantic's model_json_schema method.

The language model also doesn't need to stream anything when structured output is enabled. Thus, standard invocation will be performed regardless of whether the event_emitter parameter is provided or not.

When enabled, the structured output is stored in the structured_output attribute in the output. 1. If the schema is a JSON schema dictionary, the structured output is a dictionary. 2. If the schema is a Pydantic BaseModel class, the structured output is a Pydantic model.

Example 1: Using a JSON schema dictionary

Usage example:

schema = {
    "title": "Animal",
    "description": "A description of an animal.",
    "properties": {
        "color": {"title": "Color", "type": "string"},
        "name": {"title": "Name", "type": "string"},
    },
    "required": ["name", "color"],
    "type": "object",
}
lm_invoker = OpenAILMInvoker(..., response_schema=schema)

Output example:

LMOutput(structured_output={"name": "Golden retriever", "color": "Golden"})

Example 2: Using a Pydantic BaseModel class

Usage example:

class Animal(BaseModel):
    name: str
    color: str

lm_invoker = OpenAILMInvoker(..., response_schema=Animal)

Output example:

LMOutput(structured_output=Animal(name="Golden retriever", color="Golden"))

Analytics tracking

Analytics tracking is a feature that allows the module to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True. When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"status": "completed", "incomplete_details": {"reason": None}},
)

Retry and timeout

The OpenAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=0.0)  # No retry, no timeout
retry_config = RetryConfig(max_retries=0, timeout=10.0)  # No retry, 10.0 seconds timeout
retry_config = RetryConfig(max_retries=5, timeout=0.0)  # 5 max retries, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = OpenAILMInvoker(..., retry_config=retry_config)

Reasoning

OpenAI's o-series models are classified as reasoning models. Reasoning models think before they answer, producing a long internal chain of thought before responding to the user. Reasoning models excel in complex problem solving, coding, scientific reasoning, and multi-step planning for agentic workflows.

The reasoning effort of reasoning models can be set via the reasoning_effort parameter. This parameter will guide the models on how many reasoning tokens it should generate before creating a response to the prompt. Available options include: 1. "low": Favors speed and economical token usage. 2. "medium": Favors a balance between speed and reasoning accuracy. 3. "high": Favors more complete reasoning at the cost of more tokens generated and slower responses. When not set, the reasoning effort will be equivalent to medium by default.

OpenAI doesn't expose the raw reasoning tokens. However, the summary of the reasoning tokens can still be generated. The summary level can be set via the reasoning_summary parameter. Available options include: 1. "auto": The model decides the summary level automatically. 2. "detailed": The model will generate a detailed summary of the reasoning tokens. Reasoning summary is not compatible with tool calling. When enabled, the reasoning summary will be stored in the reasoning attribute in the output.

Output example:

LMOutput(
    response="Golden retriever is a good dog breed.",
    reasoning=[Reasoning(id="x", reasoning="Let me think about it...")],
)

When streaming is enabled along with reasoning summary, the reasoning summary token will be streamed with the EventType.DATA event type.

Streaming output example:

{"type": "data", "value": '{"data_type": "thinking_start", "data_value": ""}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "Let me think "}', ...}
{"type": "data", "value": '{"data_type": "thinking", "data_value": "about it..."}', ...}
{"type": "data", "value": '{"data_type": "thinking_end", "data_value": ""}', ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Setting reasoning-related parameters for non-reasoning models will raise an error.

Code interpreter

The code interpreter is a feature that allows the language model to write and run Python code in a sandboxed environment to solve complex problems in domains like data analysis, coding, and math. This feature can be enabled by setting the code_interpreter parameter to True.

Usage example:

lm_invoker = OpenAILMInvoker(..., code_interpreter=True)

When code interpreter is enabled, it is highly recommended to instruct the model to use the "python tool" in the system message, as "python tool" is the term recognized by the model to refer to the code interpreter.

Prompt example:

prompt = [
    ("system", ["You are a data analyst. Use the python tool to generate a file."]),
    ("user", ["Show an histogram of the following data: [1, 2, 1, 4, 1, 2, 4, 2, 3, 1]"]),
]

When code interpreter is enabled, the code execution results are stored in the code_exec_results attribute in the output.

Output example:

LMOutput(
    response="The histogram is attached.",
    code_exec_results=[
        CodeExecResult(
            id="123",
            code="import matplotlib.pyplot as plt...",
            output=[Attachment(data=b"...", mime_type="image/png")],
        ),
    ],
)

When streaming is enabled, the executed code will be streamed with the EventType.DATA event type. Streaming output example:

{"type": "data", "value": '{"data_type": "code_start", "data_value": ""}', ...}
{"type": "data", "value": '{"data_type": "code", "data_value": "import matplotlib"}', ...}
{"type": "data", "value": '{"data_type": "code", "data_value": ".pyplot as plt..."}', ...}
{"type": "data", "value": '{"data_type": "code_end", "data_value": ""}', ...}
{"type": "response", "value": "The histogram ", ...}
{"type": "response", "value": "is attached.", ...}

Web search

The web search is a feature that allows the language model to search the web for relevant information. This feature can be enabled by setting the web_search parameter to True.

Usage example:

lm_invoker = OpenAILMInvoker(..., web_search=True)

When web search is enabled, the language model will search the web for relevant information and may cite the relevant sources. The citations will be stored as Chunk objects in the citations attribute in the output. The content of the Chunk object is the type of the citation, e.g. "url_citation".

Output example:

LMOutput(
    response="The winner of the match is team A ([Example title](https://www.example.com)).",
    citations=[
        Chunk(
            id="123",
            content="url_citation",
            metadata={
                "start_index": 164,
                "end_index": 275,
                "title": "Example title",
                "url": "https://www.example.com",
                "type": "url_citation",
            },
        ),
    ],
)

When streaming is enabled, the web search activities will be streamed with the EventType.DATA event type. Streaming output example:

{"type": "data", "value": '{"data_type": "activity", "data_value": "{\"query\": \"search query\"}", ...}', ...}
{"type": "response", "value": "The winner of the match ", ...}
{"type": "response", "value": "is team A ([Example title](https://www.example.com)).", ...}

Output types

The output of the OpenAILMInvoker is of type MultimodalOutput, which is a type alias that can represent: 1. str: The text response if no additional output is needed. 2. LMOutput: A Pydantic model with the following attributes if any additional output is needed: 2.1. response (str): The text response. 2.2. tool_calls (list[ToolCall]): The tool calls, if the tools parameter is defined and the language model decides to invoke tools. Defaults to an empty list. 2.3. structured_output (dict[str, Any] | BaseModel | None): The structured output, if the response_schema parameter is defined. Defaults to None. 2.4. token_usage (TokenUsage | None): The token usage analytics, if the output_analytics parameter is set to True. Defaults to None. 2.5. duration (float | None): The duration of the invocation in seconds, if the output_analytics parameter is set to True. Defaults to None. 2.6. finish_details (dict[str, Any] | None): The details about how the generation finished, if the output_analytics parameter is set to True. Defaults to None. 2.7. reasoning (list[Reasoning]): The reasoning objects, if the reasoning_summary parameter is provided for reasoning models. Defaults to an empty list. 2.8. citations (list[Chunk]): The citations, if the web_search is enabled and the language model decides to cite the relevant sources. Defaults to an empty list. 2.9. code_exec_results (list[CodeExecResult]): The code execution results, if the code interpreter is enabled and the language model decides to execute any codes. Defaults to an empty list.

Initializes a new instance of the OpenAILMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the OpenAI model.	required
`api_key`	`str \| None`	The API key for authenticating with OpenAI. Defaults to None, in which case the `OPENAI_API_KEY` environment variable will be used.	`None`
`model_kwargs`	`dict[str, Any] \| None`	Additional model parameters. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool] \| None`	Tools provided to the language model to enable tool calling. Defaults to None.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	`None`
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	`None`
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated. Defaults to None.	`None`
`code_interpreter`	`bool`	Whether to enable the code interpreter. Defaults to False.	`False`
`web_search`	`bool`	Whether to enable the web search. Defaults to False.	`False`
`bind_tools_params`	`dict[str, Any] \| None`	Deprecated parameter to add tool calling capability. If provided, must at least include the `tools` key that is equivalent to the `tools` parameter. Retained for backward compatibility. Defaults to None.	`None`
`with_structured_output_params`	`dict[str, Any] \| None`	Deprecated parameter to instruct the model to produce output with a certain schema. If provided, must at least include the `schema` key that is equivalent to the `response_schema` parameter. Retained for backward compatibility. Defaults to None.	`None`

Raises:

Type	Description
`ValueError`

`set_response_schema(response_schema)`

Sets the response schema for the OpenAI language model.

This method sets the response schema for the OpenAI language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required

`TGILMInvoker(url, username='', password='', api_key='', default_hyperparameters=None)`

Bases: OpenAICompatibleLMInvoker

A language model invoker to interact with language models hosted in Text Generation Inference (TGI).

This class has been deprecated as Text Generation Inference is now supported through OpenAICompatibleLMInvoker. This class is maintained for backward compatibility and will be removed in version 0.5.0.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client`	`AsyncOpenAI`	The OpenAI client instance.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Any]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.

Initializes a new instance of the TGILMInvoker class.

Parameters:

Name	Type	Description	Default
`url`	`str`	The URL of the TGI service.	required
`username`	`str`	The username for Basic Authentication. Defaults to an empty string.	`''`
`password`	`str`	The password for Basic Authentication. Defaults to an empty string.	`''`
`api_key`	`str`	The API key for authenticating with the TGI service. Defaults to an empty string.	`''`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`