Lm Invoker

Modules concerning the language model invokers used in Gen AI applications.

AnthropicLMInvoker(model_name, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, thinking=False, thinking_budget=DEFAULT_THINKING_BUDGET, simplify_events=False)

Bases: BaseLMInvoker

A language model invoker to interact with Anthropic language models.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client	AsyncAnthropic	The Anthropic client instance.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	Tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig	The retry configuration for the language model.
thinking	bool	Whether to enable thinking. Only allowed for thinking models.
thinking_budget	int	The tokens allocated for the thinking process. Only allowed for thinking models.

Basic usage

The AnthropicLMInvoker can be used as follows:

lm_invoker = AnthropicLMInvoker(model_name="claude-sonnet-4-20250514")
result = await lm_invoker.invoke("Hi there!")

Input types

The AnthropicLMInvoker supports the following input types: text, image, and document. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The AnthropicLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The AnthropicLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = AnthropicLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = AnthropicLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

Structured output is not compatible with tool calling or thinking. When structured output is enabled, streaming is disabled.

Tool calling

The AnthropicLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = AnthropicLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Thinking

The AnthropicLMInvoker can be configured to perform step-by-step thinking process before answering. This feature can be enabled by setting the thinking parameter to True.

Thinking outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = AnthropicLMInvoker(..., thinking=True, thinking_budget=1024)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

The amount of tokens allocated for the thinking process can be set via the thinking_budget parameter. For more information, please refer to the following documentation: https://docs.claude.com/en/docs/build-with-claude/extended-thinking#working-with-thinking-budgets.

Thinking is only available for certain models, starting from Claude Sonnet 3.7.

Analytics tracking

The AnthropicLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The AnthropicLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = AnthropicLMInvoker(..., retry_config=retry_config)

Batch processing

The AnthropicLMInvoker supports batch processing, which allows the language model to process multiple requests in a single call. Batch processing is supported through the batch attribute.

Usage example:

requests = {"request_1": "What color is the sky?", "request_2": "What color is the grass?"}
results = await lm_invoker.batch.invoke(requests)

Output example:

{
    "request_1": LMOutput(outputs=[LMOutputItem(type="text", output="The sky is blue.")]),
    "request_2": LMOutput(finish_details={"type": "error", "error": {"message": "...", ...}, ...}),
}

The AnthropicLMInvoker also supports the following standalone batch processing operations:

1. Create a batch job: python requests = {"request_1": "What color is the sky?", "request_2": "What color is the grass?"} batch_id = await lm_invoker.batch.create(requests)

2. Get the status of a batch job: python status = await lm_invoker.batch.status(batch_id)

3. Retrieve the results of a batch job: python results = await lm_invoker.batch.retrieve(batch_id)

   Output example: python { "request_1": LMOutput(outputs=[LMOutputItem(type="text", output="The sky is blue.")]), "request_2": LMOutput(finish_details={"type": "error", "error": {"message": "...", ...}, ...}), }

4. List the batch jobs: python batch_jobs = await lm_invoker.batch.list()

   Output example: python [ {"id": "batch_123", "status": "finished"}, {"id": "batch_456", "status": "in_progress"}, {"id": "batch_789", "status": "canceling"}, ]

5. Cancel a batch job: python await lm_invoker.batch.cancel(batch_id)

Initializes the AnthropicLmInvoker instance.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the Anthropic language model.	required
api_key	str | None	The Anthropic API key. Defaults to None, in which case the ANTHROPIC_API_KEY environment variable will be used.	None
model_kwargs	dict[str, Any] | None	Additional keyword arguments for the Anthropic client.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None, in which case an empty list is used.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
thinking	bool	Whether to enable thinking. Only allowed for thinking models. Defaults to False.	False
thinking_budget	int	The tokens allocated for the thinking process. Must be greater than or equal to 1024. Only allowed for thinking models. Defaults to DEFAULT_THINKING_BUDGET.	DEFAULT_THINKING_BUDGET
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

Raises:

Type	Description
ValueError

set_response_schema(response_schema)

Sets the response schema for the Anthropic language model.

This method sets the response schema for the Anthropic language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

Raises:

Type	Description
ValueError	If tools exists.

set_tools(tools)

Sets the tools for the Anthropic language model.

This method sets the tools for the Anthropic language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool | Tool]	The list of tools to be used.	required

Raises:

Type	Description
ValueError	If response_schema exists.

AzureOpenAILMInvoker(azure_endpoint, azure_deployment, api_key=None, api_version=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, reasoning_summary=None, simplify_events=False)

Bases: OpenAILMInvoker

A language model invoker to interact with Azure OpenAI language models.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the Azure OpenAI language model deployment.
client_kwargs	dict[str, Any]	The keyword arguments for the Azure OpenAI client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig	The retry configuration for the language model.
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort.
reasoning_summary	ReasoningSummary | None	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated.
mcp_servers	list[MCPServer]	The list of MCP servers to enable MCP tool calling.
code_interpreter	bool	Whether to enable the code interpreter. Currently not supported.
web_search	bool	Whether to enable the web search. Currently not supported.

Basic usage

The AzureOpenAILMInvoker can be used as follows:

lm_invoker = AzureOpenAILMInvoker(
    azure_endpoint="https://<your-azure-openai-endpoint>.openai.azure.com/openai/v1",
    azure_deployment="<your-azure-openai-deployment>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The AzureOpenAILMInvoker supports the following input types: text, document, and image. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The AzureOpenAILMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The AzureOpenAILMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = AzureOpenAILMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = AzureOpenAILMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Tool calling

The AzureOpenAILMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = AzureOpenAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Reasoning

The AzureOpenAILMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as GPT-5 models and o-series models.

The reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include minimal, low, medium, and high.

While the raw reasoning tokens are not available, the summary of the reasoning tokens can still be generated. This can be done by passing the desired summary level via the reasoning_summary parameter. Available options include auto and detailed.

Reasoning summaries are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = AzureOpenAILMInvoker(..., reasoning_effort="high", reasoning_summary="detailed")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Reasoning summary is not compatible with tool calling.

Analytics tracking

The AzureOpenAILMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The AzureOpenAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = AzureOpenAILMInvoker(..., retry_config=retry_config)

Initializes a new instance of the AzureOpenAILMInvoker class.

Parameters:

Name	Type	Description	Default
azure_endpoint	str	The endpoint of the Azure OpenAI service.	required
azure_deployment	str	The deployment name of the Azure OpenAI service.	required
api_key	str | None	The API key for authenticating with Azure OpenAI. Defaults to None, in which case the AZURE_OPENAI_API_KEY environment variable will be used.	None
api_version	str | None	Deprecated parameter to be removed in v0.6. Defaults to None.	None
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	None
reasoning_summary	ReasoningSummary | None	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated. Defaults to None.	None
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

Raises:

Type	Description
ValueError

BedrockLMInvoker(model_name, access_key_id=None, secret_access_key=None, region_name='us-east-1', model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None)

Bases: BaseLMInvoker

A language model invoker to interact with AWS Bedrock language models.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
session	Session	The Bedrock client session.
client_kwargs	dict[str, Any]	The Bedrock client kwargs.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	Tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig	The retry configuration for the language model.

Basic usage

The BedrockLMInvoker can be used as follows:

lm_invoker = BedrockLMInvoker(
    model_name="us.anthropic.claude-sonnet-4-20250514-v1:0",
    aws_access_key_id="<your-aws-access-key-id>",
    aws_secret_access_key="<your-aws-secret-access-key>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The BedrockLMInvoker supports the following input types: text, document, image, and video. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The BedrockLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The BedrockLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = BedrockLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = BedrockLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

Structured output is not compatible with tool calling. When structured output is enabled, streaming is disabled.

Tool calling

The BedrockLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = BedrockLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Analytics tracking

The BedrockLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The BedrockLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = BedrockLMInvoker(..., retry_config=retry_config)

Initializes the BedrockLMInvoker instance.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the Bedrock language model.	required
access_key_id	str | None	The AWS access key ID. Defaults to None, in which case the AWS_ACCESS_KEY_ID environment variable will be used.	None
secret_access_key	str | None	The AWS secret access key. Defaults to None, in which case the AWS_SECRET_ACCESS_KEY environment variable will be used.	None
region_name	str	The AWS region name. Defaults to "us-east-1".	'us-east-1'
model_kwargs	dict[str, Any] | None	Additional keyword arguments for the Bedrock client.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None

Raises:

Type	Description
ValueError	If response_schema is provided, but tools are also provided.
ValueError	If access_key_id or secret_access_key is neither provided nor set in the AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY environment variables, respectively.

set_response_schema(response_schema)

Sets the response schema for the Bedrock language model.

This method sets the response schema for the Bedrock language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

Raises:

Type	Description
ValueError	If tools exists.

set_tools(tools)

Sets the tools for the Bedrock language model.

This method sets the tools for the Bedrock language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool | Tool]	The list of tools to be used.	required

Raises:

Type	Description
ValueError	If response_schema exists.

DatasaurLMInvoker(base_url, api_key=None, model_kwargs=None, default_hyperparameters=None, output_analytics=False, retry_config=None, citations=False)

Bases: OpenAIChatCompletionsLMInvoker

A language model invoker to interact with Datasaur LLM Projects Deployment API.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_kwargs	dict[str, Any]	The keyword arguments for the OpenAI client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Any]	The list of tools provided to the model to enable tool calling. Currently not supported.
response_schema	ResponseSchema | None	The schema of the response. Currently not supported.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.
citations	bool	Whether to output the citations.

Basic usage

The DatasaurLMInvoker can be used as follows:

lm_invoker = DatasaurLMInvoker(base_url="https://deployment.datasaur.ai/api/deployment/teamId/deploymentId/")
result = await lm_invoker.invoke("Hi there!")

Input types

The DatasaurLMInvoker supports the following input types: text, audio, image, and document. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The DatasaurLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Citations

The DatasaurLMInvoker can be configured to output the citations used to generate the response. This feature can be enabled by setting the citations parameter to True.

Citations outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the citations property.

Usage example:

lm_invoker = DatasaurLMInvoker(..., citations=True)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="citation", output=Chunk(id="123", content="...", metadata={...}, score=0.95)),
        LMOutputItem(type="text", output="According to recent reports... ([Source](https://www.example.com))."),
    ],
)

Analytics tracking

The DatasaurLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The DatasaurLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = DatasaurLMInvoker(..., retry_config=retry_config)

Initializes a new instance of the DatasaurLMInvoker class.

Parameters:

Name	Type	Description	Default
base_url	str	The base URL of the Datasaur LLM Projects Deployment API.	required
api_key	str | None	The API key for authenticating with Datasaur LLM Projects Deployment API. Defaults to None, in which case the DATASAUR_API_KEY environment variable will be used.	None
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
citations	bool	Whether to output the citations. Defaults to False.	False

Raises:

Type	Description
ValueError	If the api_key is not provided and the DATASAUR_API_KEY environment variable is not set.

set_response_schema(response_schema)

Sets the response schema for the Datasaur LLM Projects Deployment API.

This method is raises a NotImplementedError because the Datasaur LLM Projects Deployment API does not support response schema.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

Raises:

Type	Description
NotImplementedError	This method is not supported for the Datasaur LLM Projects Deployment API.

set_tools(tools)

Sets the tools for the Datasaur LLM Projects Deployment API.

This method is raises a NotImplementedError because the Datasaur LLM Projects Deployment API does not support tools.

Parameters:

Name	Type	Description	Default
tools	list[Tool | Tool]	The list of tools to be used.	required

Raises:

Type	Description
NotImplementedError	This method is not supported for the Datasaur LLM Projects Deployment API.

GoogleLMInvoker(model_name, api_key=None, credentials_path=None, project_id=None, location='us-central1', model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, thinking=None, thinking_budget=DEFAULT_THINKING_BUDGET, simplify_events=False)

Bases: BaseLMInvoker

A language model invoker to interact with Google language models.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_params	dict[str, Any]	The Google client instance init parameters.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Any]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.
thinking	bool	Whether to enable thinking. Only allowed for thinking models.
thinking_budget	int	The tokens allowed for thinking process. Only allowed for thinking models. If set to -1, the model will control the budget automatically.
image_generation	bool	Whether to generate image. Only allowed for image generation models.

Basic usage

The GoogleLMInvoker can be used as follows:

lm_invoker = GoogleLMInvoker(model_name="gemini-2.5-flash")
result = await lm_invoker.invoke("Hi there!")

Authentication

The GoogleLMInvoker can use either Google Gen AI or Google Vertex AI.

Google Gen AI is recommended for quick prototyping and development. It requires a Gemini API key for authentication.

Usage example:

lm_invoker = GoogleLMInvoker(
    model_name="gemini-2.5-flash",
    api_key="your_api_key"
)

Google Vertex AI is recommended to build production-ready applications. It requires a service account JSON file for authentication.

Usage example:

lm_invoker = GoogleLMInvoker(
    model_name="gemini-2.5-flash",
    credentials_path="path/to/service_account.json"
)

If neither api_key nor credentials_path is provided, Google Gen AI will be used by default. The GOOGLE_API_KEY environment variable will be used for authentication.

Input types

The GoogleLMInvoker supports the following input types: text, audio, document, image, and video. Non-text inputs can be passed as an Attachment object with either the user or assistant role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The GoogleLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The GoogleLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = GoogleLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = GoogleLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

Structured output is not compatible with tool calling. When structured output is enabled, streaming is disabled.

Image generation

The GoogleLMInvoker can be configured to generate images. This feature can be enabled by using an image generation model, such as gemini-2.5-flash-image.

Image outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the attachments property.

Usage example:

lm_invoker = GoogleLMInvoker("gemini-2.5-flash-image")
result = await lm_invoker.invoke("Create a picture...")
result.attachments[0].write_to_file("path/to/local/image.png")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="Creating a picture..."),
        LMOutputItem(
            type="attachment",
            output=Attachment(filename="image.png", mime_type="image/png", data=b"..."),
        ),
    ],
)

Image generation is not compatible with tool calling and thinking. When image generation is enabled, streaming is disabled.

Tool calling

The GoogleLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = GoogleLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Thinking

The GoogleLMInvoker can be configured to perform step-by-step thinking process before answering. This feature can be enabled by setting the thinking parameter to True.

Thinking outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = GoogleLMInvoker(..., thinking=True, thinking_budget=1024)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

The amount of tokens allocated for the thinking process can be set via the thinking_budget parameter. For more information, please refer to the following documentation: https://ai.google.dev/gemini-api/docs/thinking

Thinking is only available for certain models, starting from Gemini 2.5 series. Thinking is required for Gemini 2.5 Pro models.

Analytics tracking

The GoogleLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The GoogleLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = GoogleLMInvoker(..., retry_config=retry_config)

Batch processing

The GoogleLMInvoker supports batch processing, which allows the language model to process multiple requests in a single call. Batch processing is supported through the batch attribute.

Due to Google SDK limitations with batch processing: 1. Only inline requests are currently supported (not file-based or BigQuery sources). 2. The total size of all requests must be under 20MB. 3. Original request indices are not preserved in the results. The results are keyed by request index in the format '1', '2', etc, in which order are preserved based on the original request order. If you want to use custom request IDs, you can pass them as a list of strings to the custom_request_ids keyword argument

Usage example:

requests = {"1": "What color is the sky?", "2": "What color is the grass?"}
results = await lm_invoker.batch.invoke(requests)

Output example:

{
    "1": LMOutput(outputs=[LMOutputItem(type="text", output="The sky is blue.")]),
    "2": LMOutput(finish_details={"type": "error", "message": "..."}),
}

The GoogleLMInvoker also supports the following standalone batch processing operations:

1. Create a batch job: python requests = {"1": "What color is the sky?", "2": "What color is the grass?"} batch_id = await lm_invoker.batch.create(requests)

2. Get the status of a batch job: python status = await lm_invoker.batch.status(batch_id)

3. Retrieve the results of a batch job:

   In default, the results will be keyed by request index in the format '1', '2', etc, in which order are preserved based on the original request order.

   python results = await lm_invoker.batch.retrieve(batch_id)

   Output example: python { "1": LMOutput(outputs=[LMOutputItem(type="text", output="The sky is blue.")]), "2": LMOutput(finish_details={"type": "error", "error": {"message": "...", ...}, ...}), }

   If you pass custom_request_ids to the create method, the results will be keyed by the custom_request_ids. python results = await lm_invoker.batch.retrieve(batch_id, custom_request_ids=["request_1", "request_2"])

   Output example: python { "request_1": LMOutput(outputs=[LMOutputItem(type="text", output="The sky is blue.")]), "request_2": LMOutput(finish_details={"type": "error", "error": {"message": "...", ...}, ...}), }

4. List the batch jobs: python batch_jobs = await lm_invoker.batch.list()

   Output example: python [ {"id": "batch_123", "status": "finished"}, {"id": "batch_456", "status": "in_progress"}, {"id": "batch_789", "status": "canceling"}, ]

5. Cancel a batch job: python await lm_invoker.batch.cancel(batch_id)

Initializes a new instance of the GoogleLMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the model to use.	required
api_key	str | None	Required for Google Gen AI authentication. Cannot be used together with credentials_path. Defaults to None.	None
credentials_path	str | None	Required for Google Vertex AI authentication. Path to the service account credentials JSON file. Cannot be used together with api_key. Defaults to None.	None
project_id	str | None	The Google Cloud project ID for Vertex AI. Only used when authenticating with credentials_path. Defaults to None, in which case it will be loaded from the credentials file.	None
location	str	The location of the Google Cloud project for Vertex AI. Only used when authenticating with credentials_path. Defaults to "us-central1".	'us-central1'
model_kwargs	dict[str, Any] | None	Additional keyword arguments for the Google Vertex AI client.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
thinking	bool | None	Whether to enable thinking. Only allowed for thinking models. Defaults to True for Gemini 2.5 Pro models and False for other models.	None
thinking_budget	int	The tokens allowed for thinking process. Only allowed for thinking models. Defaults to -1, in which case the model will control the budget automatically.	DEFAULT_THINKING_BUDGET
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

Note

If neither api_key nor credentials_path is provided, Google Gen AI will be used by default. The GOOGLE_API_KEY environment variable will be used for authentication.

set_response_schema(response_schema)

Sets the response schema for the Google language model.

This method sets the response schema for the Google language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

Raises:

Type	Description
ValueError	If tools exists.

set_tools(tools)

Sets the tools for the Google language model.

This method sets the tools for the Google language model. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool | Tool]	The list of tools to be used.	required

Raises:

Type	Description
ValueError	If response_schema exists.

LangChainLMInvoker(model=None, model_class_path=None, model_name=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None)

Bases: BaseLMInvoker

A language model invoker to interact with LangChain's BaseChatModel.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
model	BaseChatModel	The LangChain's BaseChatModel instance.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Any]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.

Basic usage

The LangChainLMInvoker can be used as follows:

lm_invoker = LangChainLMInvoker(
    model_class_path="langchain_openai.ChatOpenAI",
    model_name="gpt-5-nano",
)
result = await lm_invoker.invoke("Hi there!")

Initialization

The LangChainLMInvoker can be initialized by either passing:

1. A LangChain's BaseChatModel instance: Usage example:

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-5-nano", api_key="your_api_key")
lm_invoker = LangChainLMInvoker(model=model)

1. A model path in the format of ".": Usage example:

lm_invoker = LangChainLMInvoker(
    model_class_path="langchain_openai.ChatOpenAI",
    model_name="gpt-5-nano",
    model_kwargs={"api_key": "your_api_key"}
)

For the list of supported providers, please refer to the following table: https://python.langchain.com/docs/integrations/chat/#featured-providers

Input types

The LangChainLMInvoker supports the following input types: text and image. Non-text inputs can be passed as an Attachment object and with specific roles, depending on the language model's capabilities.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The LangChainLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The LangChainLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = LangChainLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = LangChainLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

Structured output is not compatible with tool calling. When structured output is enabled, streaming is disabled.

Tool calling

The LangChainLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = LangChainLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Analytics tracking

The LangChainLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The LangChainLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = LangChainLMInvoker(..., retry_config=retry_config)

Initializes a new instance of the LangChainLMInvoker class.

Parameters:

Name	Type	Description	Default
model	BaseChatModel | None	The LangChain's BaseChatModel instance. If provided, will take precedence over the model_class_path parameter. Defaults to None.	None
model_class_path	str | None	The LangChain's BaseChatModel class path. Must be formatted as "." (e.g. "langchain_openai.ChatOpenAI"). Ignored if model is provided. Defaults to None.	None
model_name	str | None	The model name. Only used if model_class_path is provided. Defaults to None.	None
model_kwargs	dict[str, Any] | None	The additional keyword arguments. Only used if model_class_path is provided. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None

Raises:

Type	Description
ValueError	If response_schema is provided, but tools are also provided.

set_response_schema(response_schema)

Sets the response schema for the LangChain's BaseChatModel.

This method sets the response schema for the LangChain's BaseChatModel. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

Raises:

Type	Description
ValueError	If tools exists.

set_tools(tools)

Sets the tools for LangChain's BaseChatModel.

This method sets the tools for LangChain's BaseChatModel. Any existing tools will be replaced.

Parameters:

Name	Type	Description	Default
tools	list[Tool]	The list of tools to be used.	required

Raises:

Type	Description
ValueError	If response_schema exists.

LiteLLMLMInvoker(model_id, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, simplify_events=False)

Bases: OpenAIChatCompletionsLMInvoker

A language model invoker to interact with language models using LiteLLM.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
completion	function	The LiteLLM's completion function.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.

Basic usage

The LiteLLMLMInvoker can be used as follows:

lm_invoker = LiteLLMLMInvoker(model_id="openai/gpt-5-nano")
result = await lm_invoker.invoke("Hi there!")

Initialization

The LiteLLMLMInvoker provides an interface to interact with multiple language model providers. In order to use this class: 1. The model_id parameter must be in the format of provider/model_name. e.g. openai/gpt-4o-mini. 2. The required credentials must be provided via the environment variables.

Usage example:

os.environ["OPENAI_API_KEY"] = "your_openai_api_key"
lm_invoker = LiteLLMLMInvoker(model_id="openai/gpt-4o-mini")

For the complete list of supported providers and their required credentials, please refer to the LiteLLM documentation: https://docs.litellm.ai/docs/providers/

Input types

The LiteLLMLMInvoker supports the following input types: text, audio, and image. Non-text inputs can be passed as a Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The LiteLLMLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The LiteLLMLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = LiteLLMLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = LiteLLMLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Tool calling

The LiteLLMLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = LiteLLMLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Reasoning

The LiteLLMLMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as GPT-5 models and o-series models.

The reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include minimal, low, medium, and high.

Some models may also output the reasoning tokens. In this case, the reasoning tokens are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example: python {"type": "thinking_start", "value": "", ...} {"type": "thinking", "value": "I'm ", ...} {"type": "thinking", "value": "thinking...", ...} {"type": "thinking_end", "value": "", ...} {"type": "response", "value": "Golden retriever ", ...} {"type": "response", "value": "is a good dog breed.", ...} Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Setting reasoning-related parameters for non-reasoning models will raise an error.

Analytics tracking

The LiteLLMLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

When streaming is enabled, token usage is not supported.

Retry and timeout

The LiteLLMLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = LiteLLMLMInvoker(..., retry_config=retry_config)

Initializes a new instance of the LiteLLMLMInvoker class.

Parameters:

Name	Type	Description	Default
model_id	str	The ID of the model to use. Must be in the format of provider/model_name.	required
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Defaults to None.	None
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

OpenAIChatCompletionsLMInvoker(model_name, api_key=None, base_url=OPENAI_DEFAULT_URL, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, simplify_events=False)

Bases: BaseLMInvoker

A language model invoker to interact with OpenAI language models using the Chat Completions API.

This class provides support for OpenAI's Chat Completions API schema. Use this class only when you have a specific reason to use the Chat Completions API over the Responses API, as OpenAI recommends using the Responses API whenever possible. The Responses API schema is supported through the OpenAILMInvoker class.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_kwargs	dict[str, Any]	The keyword arguments for the OpenAI client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.

Basic usage

The OpenAIChatCompletionsLMInvoker can be used as follows:

lm_invoker = OpenAIChatCompletionsLMInvoker(model_name="gpt-5-nano")
result = await lm_invoker.invoke("Hi there!")

OpenAI compatible endpoints

The OpenAIChatCompletionsLMInvoker can also be used to interact with endpoints that are compatible with OpenAI's Chat Completions API schema. This includes but are not limited to: 1. DeepInfra (https://deepinfra.com/) 2. DeepSeek (https://deepseek.com/) 3. Groq (https://groq.com/) 4. OpenRouter (https://openrouter.ai/) 5. Text Generation Inference (https://github.com/huggingface/text-generation-inference) 6. Together.ai (https://together.ai/) 7. vLLM (https://vllm.ai/) Please note that the supported features and capabilities may vary between different endpoints and language models. Using features that are not supported by the endpoint will result in an error.

This customization can be done by setting the base_url parameter to the base URL of the endpoint:

lm_invoker = OpenAIChatCompletionsLMInvoker(
    model_name="llama3-8b-8192",
    api_key="<your-api-key>",
    base_url="https://api.groq.com/openai/v1",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The OpenAIChatCompletionsLMInvoker supports the following input types: text, audio, document, and image. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The OpenAIChatCompletionsLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The OpenAIChatCompletionsLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = OpenAIChatCompletionsLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = OpenAIChatCompletionsLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Tool calling

The OpenAIChatCompletionsLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = OpenAIChatCompletionsLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Reasoning

The OpenAILMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as GPT-5 models and o-series models.

The reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include minimal, low, medium, and high.

Some models may also output the reasoning tokens. In this case, the reasoning tokens are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example: python {"type": "thinking_start", "value": "", ...} {"type": "thinking", "value": "I'm ", ...} {"type": "thinking", "value": "thinking...", ...} {"type": "thinking_end", "value": "", ...} {"type": "response", "value": "Golden retriever ", ...} {"type": "response", "value": "is a good dog breed.", ...} Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Setting reasoning-related parameters for non-reasoning models will raise an error.

Analytics tracking

The OpenAIChatCompletionsLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

When streaming is enabled, token usage is not supported.

Retry and timeout

The OpenAIChatCompletionsLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = OpenAIChatCompletionsLMInvoker(..., retry_config=retry_config)

Initializes a new instance of the OpenAIChatCompletionsLMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the OpenAI model.	required
api_key	str | None	The API key for authenticating with OpenAI. Defaults to None, in which case the OPENAI_API_KEY environment variable will be used. If the endpoint does not require an API key, a dummy value can be passed (e.g. "").	None
base_url	str	The base URL of a custom endpoint that is compatible with OpenAI's Chat Completions API schema. Defaults to OpenAI's default URL.	OPENAI_DEFAULT_URL
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
reasoning_effort	str | None	The reasoning effort for the language model. Defaults to None.	None
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

set_response_schema(response_schema)

Sets the response schema for the OpenAI language model.

This method sets the response schema for the OpenAI language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

OpenAICompatibleLMInvoker(model_name, base_url, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, simplify_events=False)

Bases: OpenAIChatCompletionsLMInvoker

A language model invoker to interact with endpoints compatible with OpenAI's chat completion API contract.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_kwargs	dict[str, Any]	The keyword arguments for the OpenAI client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.

This class is deprecated and will be removed in v0.6. Please use the OpenAIChatCompletionsLMInvoker class instead.

Initializes a new instance of the OpenAICompatibleLMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the language model hosted on the OpenAI compatible endpoint.	required
base_url	str	The base URL for the OpenAI compatible endpoint.	required
api_key	str | None	The API key for authenticating with the OpenAI compatible endpoint. Defaults to None, in which case the OPENAI_API_KEY environment variable will be used. If the endpoint does not require an API key, a dummy value can be passed (e.g. "").	None
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
reasoning_effort	str | None	The reasoning effort for the language model. Defaults to None.	None
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

OpenAILMInvoker(model_name, api_key=None, base_url=OPENAI_DEFAULT_URL, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, reasoning_summary=None, image_generation=False, mcp_servers=None, code_interpreter=False, web_search=False, simplify_events=False)

Bases: BaseLMInvoker

A language model invoker to interact with OpenAI language models.

This class provides support for OpenAI's Responses API schema, which is recommended by OpenAI as the preferred API to use whenever possible. Use this class unless you have a specific reason to use the Chat Completions API instead. The Chat Completions API schema is supported through the OpenAIChatCompletionsLMInvoker class.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_kwargs	dict[str, Any]	The keyword arguments for the OpenAI client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig	The retry configuration for the language model.
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort.
reasoning_summary	ReasoningSummary | None	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated.
image_generation	bool	Whether to enable image generation.
mcp_servers	list[MCPServer]	The list of MCP servers to enable MCP tool calling.
code_interpreter	bool	Whether to enable the code interpreter.
web_search	bool	Whether to enable the web search.

Basic usage

The OpenAILMInvoker can be used as follows:

lm_invoker = OpenAILMInvoker(model_name="gpt-5-nano")
result = await lm_invoker.invoke("Hi there!")

OpenAI compatible endpoints

The OpenAILMInvoker can also be used to interact with endpoints that are compatible with OpenAI's Responses API schema. This includes but are not limited to: 1. SGLang (https://github.com/sgl-project/sglang) Please note that the supported features and capabilities may vary between different endpoints and language models. Using features that are not supported by the endpoint will result in an error.

This customization can be done by setting the base_url parameter to the base URL of the endpoint:

lm_invoker = OpenAILMInvoker(
    model_name="<model-name>",
    api_key="<your-api-key>",
    base_url="<https://base-url>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The OpenAILMInvoker supports the following input types: text, document, and image. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The OpenAILMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The OpenAILMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = OpenAILMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = OpenAILMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Image generation

The OpenAILMInvoker can be configured to generate images. This feature can be enabled by setting the image_generation parameter to True.

Image outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the attachments property.

Usage example:

lm_invoker = OpenAILMInvoker(..., image_generation=True)
result = await lm_invoker.invoke("Create a picture...")
result.attachments[0].write_to_file("path/to/local/image.png")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(
            type="attachment",
            output=Attachment(filename="image.png", mime_type="image/png", data=b"..."),
        ),
    ],
)

When image generation is enabled, streaming is disabled. Image generation is only available for certain models.

Tool calling

The OpenAILMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = OpenAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

MCP tool calling

The OpenAILMInvoker can be configured to call MCP tools to perform certain tasks. This feature can be enabled by providing a list of MCP servers to the mcp_servers parameter.

MCP calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the mcp_calls property.

Usage example:

from gllm_inference.schema import MCPServer

mcp_server_1 = MCPServer(url="https://mcp_server_1.com", name="mcp_server_1")
lm_invoker = OpenAILMInvoker(..., mcp_servers=[mcp_server_1])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using MCP tools..."),
        LMOutputItem(
            type="mcp_call",
            output=MCPCall(
                id="123",
                server_name="mcp_server_1",
                tool_name="mcp_tool_1",
                args={"key": "value"},
                output="The result is 10."
            ),
        ),
    ],
)

Streaming output example:

{"type": "activity", "value": {"type": "mcp_list_tools", ...}, ...}
{"type": "activity", "value": {"type": "mcp_call", ...}, ...}
{"type": "response", "value": "The result ", ...}
{"type": "response", "value": "is 10.", ...}

Note: By default, the activity token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Reasoning

The OpenAILMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as GPT-5 models and o-series models.

The reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include minimal, low, medium, and high.

While the raw reasoning tokens are not available, the summary of the reasoning tokens can still be generated. This can be done by passing the desired summary level via the reasoning_summary parameter. Available options include auto and detailed.

Reasoning summaries are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = OpenAILMInvoker(..., reasoning_effort="high", reasoning_summary="detailed")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Reasoning summary is not compatible with tool calling.

Code interpreter

The OpenAILMInvoker can be configured to write and run Python code in a sandboxed environment. This is useful for solving complex problems in domains like data analysis, coding, and math. This feature can be enabled by setting the code_interpreter parameter to True.

When code interpreter is enabled, it is highly recommended to instruct the model to use the "python tool" in the system message, as "python tool" is the term recognized by the model to refer to the code interpreter.

Code execution results are stored in the outputs attribute of the LMOutput object and can be accessed via the code_exec_results property.

Usage example:

lm_invoker = OpenAILMInvoker(..., code_interpreter=True)
messages = [
    Message.system("You are a data analyst. Use the python tool to generate a file."]),
    Message.user("Show an histogram of the following data: [1, 2, 1, 4, 1, 2, 4, 2, 3, 1]"),
]
result = await lm_invoker.invoke(messages)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="The histogram is attached."),
        LMOutputItem(
            type="code_exec_result",
            output=CodeExecResult(
                id="123",
                code="import matplotlib.pyplot as plt...",
                output=[Attachment(data=b"...", mime_type="image/png")],
            ),
        ),
    ],
)

Streaming output example:

{"type": "code_start", "value": ""}', ...}
{"type": "code", "value": "import matplotlib"}', ...}
{"type": "code", "value": ".pyplot as plt..."}', ...}
{"type": "code_end", "value": ""}', ...}
{"type": "response", "value": "The histogram ", ...}
{"type": "response", "value": "is attached.", ...}

Note: By default, the code token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Web Search

The OpenAILMInvoker can be configured to search the web for relevant information. This feature can be enabled by setting the web_search parameter to True.

Web search citations are stored in the outputs attribute of the LMOutput object and can be accessed via the citations property.

Usage example:

lm_invoker = OpenAILMInvoker(..., web_search=True)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="citation", output=Chunk(id="123", content="...", metadata={...}, score=None)),
        LMOutputItem(type="text", output="According to recent reports... ([Source](https://example.com))."),
    ],
)

Streaming output example:

{"type": "activity", "value": {"query": "search query"}, ...}
{"type": "response", "value": "According to recent ", ...}
{"type": "response", "value": "reports... ([Source](https://example.com)).", ...}

Note: By default, the activity token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Analytics tracking

The OpenAILMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The OpenAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = OpenAILMInvoker(..., retry_config=retry_config)

Initializes a new instance of the OpenAILMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the OpenAI model.	required
api_key	str | None	The API key for authenticating with OpenAI. Defaults to None, in which case the OPENAI_API_KEY environment variable will be used. If the endpoint does not require an API key, a dummy value can be passed (e.g. "").	None
base_url	str	The base URL of a custom endpoint that is compatible with OpenAI's Responses API schema. Defaults to OpenAI's default URL.	OPENAI_DEFAULT_URL
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the model to enable tool calling. Defaults to None, in which case an empty list is used.	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	None
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	None
reasoning_summary	ReasoningSummary | None	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated. Defaults to None.	None
image_generation	bool	Whether to enable image generation. Defaults to False.	False
mcp_servers	list[MCPServer] | None	The MCP servers containing tools to be accessed by the language model. Defaults to None.	None
code_interpreter	bool	Whether to enable the code interpreter. Defaults to False.	False
web_search	bool	Whether to enable the web search. Defaults to False.	False
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

Raises:

Type	Description
ValueError

set_response_schema(response_schema)

Sets the response schema for the OpenAI language model.

This method sets the response schema for the OpenAI language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
response_schema	ResponseSchema | None	The response schema to be used.	required

PortkeyLMInvoker(model_name=None, portkey_api_key=None, provider=None, api_key=None, config=None, custom_host=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, thinking=None, thinking_budget=None, simplify_events=False)

Bases: OpenAIChatCompletionsLMInvoker

A language model invoker to interact with Portkey's Universal API.

This class provides support for Portkey’s Universal AI Gateway, which enables unified access to multiple providers (e.g., OpenAI, Anthropic, Google, Cohere, Bedrock) via a single API key. The PortkeyLMInvoker is compatible with all Portkey model routing configurations, including model catalog entries, direct providers, and pre-defined configs.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The catalog name of the language model.
client_kwargs	dict[str, Any]	The keyword arguments for the Portkey client.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig	The retry configuration for the language model.
thinking	bool	Whether to enable thinking mode for supported models.
thinking_budget	int	The maximum reasoning token budget for thinking mode.

Basic usage

The PortkeyLMInvoker supports multiple authentication methods with strict precedence order. Authentication methods are mutually exclusive and cannot be combined.

Authentication Precedence (Highest to Lowest): 1. Config ID Authentication (Highest precedence) Use a pre-configured routing setup from Portkey’s dashboard. python lm_invoker = PortkeyLMInvoker( portkey_api_key="<your-portkey-api-key>", config="pc-openai-4f6905", )

1. Model Catalog Authentication Provider name must match the provider name set in the model catalog. More details to set up the model catalog can be found in https://portkey.ai/docs/product/model-catalog#model-catalog. There are two ways to specify the model name:

2.1. Using Combined Model Name Format Specify the model_name in '@provider-name/model-name' format. python lm_invoker = PortkeyLMInvoker( portkey_api_key="<your-portkey-api-key>", model_name="@openai-custom/gpt-4o" )

2.2. Using Separate Provider and Model Name Parameters Specify the provider in '@provider-name' format and model_name separately. python lm_invoker = PortkeyLMInvoker( portkey_api_key="<your-portkey-api-key>", provider="@openai-custom", model_name="gpt-4o", )

1. Direct Provider Authentication Use the provider in 'provider-name' format and model_name parameters. python lm_invoker = PortkeyLMInvoker( portkey_api_key="<your-portkey-api-key>", provider="openai", model_name="gpt-4o", api_key="sk-...", )

Custom Host

You can also use the custom_host parameter to override the default host. This is available for all authentication methods except for Config ID authentication.

lm_invoker = PortkeyLMInvoker(..., custom_host="https://your-custom-endpoint.com")

Input types

The PortkeyLMInvoker supports text, image, document, and audio inputs. Non-text inputs can be passed as an Attachment object with the user role.

text = "What animal is in this image?"
image = Attachment.from_path("path/to/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The PortkeyLMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The PortkeyLMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = PortkeyLMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = PortkeyLMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Tool calling

The PortkeyLMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = PortkeyLMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Thinking

The PortkeyLMInvoker can be configured to perform step-by-step thinking process before answering. This feature can be enabled by setting the thinking parameter to True.

Thinking outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = PortkeyLMInvoker(..., thinking=True, thinking_budget=1024)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

The amount of tokens allocated for the thinking process can be set via the thinking_budget parameter. For more information, please refer to the following documentation: https://portkey.ai/docs/product/ai-gateway/multimodal-capabilities/thinking-mode.

Thinking is only available for certain models depending on capabilities

Analytics tracking

The PortkeyLMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

When streaming is enabled, token usage is not supported.

Retry and timeout

The PortkeyLMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = PortkeyLMInvoker(..., retry_config=retry_config)

Initializes a new instance of the PortkeyLMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str | None	The name of the model to use. Acceptable formats: 1. 'model' for direct authentication, 2. '@provider-slug/model' for model catalog authentication. Defaults to None.	None
portkey_api_key	str | None	The Portkey API key. Defaults to None, in which case the PORTKEY_API_KEY environment variable will be used.	None
provider	str | None	Provider name or catalog slug. Acceptable formats: 1. '@provider-slug' for model catalog authentication (no api_key needed), 2. 'provider' for direct authentication (requires api_key). Will be combined with model_name if model name is not in the format '@provider-slug/model'. Defaults to None.	None
api_key	str | None	Provider's API key for direct authentication. Must be used with 'provider' parameter (without '@' prefix). Not needed for catalog providers. Defaults to None.	None
config	str | None	Portkey config ID for complex routing configurations, load balancing, or fallback scenarios. Defaults to None.	None
custom_host	str | None	Custom host URL for self-hosted or custom endpoints. Can be combined with catalog providers. Defaults to None.	None
model_kwargs	dict[str, Any] | None	Additional model parameters and authentication. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for model invocation (temperature, max_tokens, etc.). Defaults to None.	None
tools	list[Tool | Tool] | None	Tools for enabling tool calling functionality. Defaults to None.	None
response_schema	ResponseSchema | None	Schema for structured output generation. Defaults to None.	None
output_analytics	bool	Whether to output detailed invocation analytics including token usage and timing. Defaults to False.	False
retry_config	RetryConfig | None	Configuration for retry behavior on failures. Defaults to None.	None
thinking	bool | None	Whether to enable thinking mode. Defaults to None.	None
thinking_budget	int | None	Thinking budget in tokens. Defaults to None.	None
simplify_events	bool	Whether to use simplified event schemas. Defaults to False.	False

XAILMInvoker(model_name, api_key=None, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, web_search=False, simplify_events=False)

Bases: BaseLMInvoker

A language model invoker to interact with xAI language models.

Attributes:

Name	Type	Description
model_id	str	The model ID of the language model.
model_provider	str	The provider of the language model.
model_name	str	The name of the language model.
client_params	dict[str, Any]	The xAI client initialization parameters.
default_hyperparameters	dict[str, Any]	Default hyperparameters for invoking the model.
tools	list[Tool]	The list of tools provided to the model to enable tool calling.
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
output_analytics	bool	Whether to output the invocation analytics.
retry_config	RetryConfig | None	The retry configuration for the language model.
reasoning_effort	ReasoningEffort | None	The reasoning effort level for reasoning models ("low" or "high").
web_search	bool	Whether to enable the web search.

Basic usage

The XAILMInvoker can be used as follows:

lm_invoker = XAILMInvoker(model_name="grok-3")
result = await lm_invoker.invoke("Hi there!")

Input types

The XAILMInvoker supports the following input types: text and image. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The XAILMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The XAILMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = XAILMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = XAILMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Tool calling

The XAILMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = XAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

Reasoning

The XAILMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as grok-3-mini.

For some models, the reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include low and high.

Some models may also output the reasoning tokens. In this case, the reasoning tokens are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = XAILMInvoker(model_name="grok-3-mini", reasoning_effort="low")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example: python {"type": "thinking_start", "value": "", ...} {"type": "thinking", "value": "I'm ", ...} {"type": "thinking", "value": "thinking...", ...} {"type": "thinking_end", "value": "", ...} {"type": "response", "value": "Golden retriever ", ...} {"type": "response", "value": "is a good dog breed.", ...} Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Web Search

The XAILMInvoker can be configured to search the web for relevant information. This feature can be enabled by setting the web_search parameter to True.

Web search citations are stored in the outputs attribute of the LMOutput object and can be accessed via the citations property.

Usage example:

lm_invoker = XAILMInvoker(..., web_search=True)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="citation", output=Chunk(id="123", content="...", metadata={...}, score=None)),
        LMOutputItem(type="text", output="According to recent reports... ([Source](https://example.com))."),
    ],
)

Analytics tracking

The XAILMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

When streaming is enabled, token usage is not supported.

Retry and timeout

The XAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = XAILMInvoker(..., retry_config=retry_config)

Initializes a new instance of the XAILMInvoker class.

Parameters:

Name	Type	Description	Default
model_name	str	The name of the xAI model.	required
api_key	str | None	The API key for authenticating with xAI. Defaults to None, in which case the XAI_API_KEY environment variable will be used.	None
model_kwargs	dict[str, Any] | None	Additional model parameters. Defaults to None.	None
default_hyperparameters	dict[str, Any] | None	Default hyperparameters for invoking the model. Defaults to None.	None
tools	list[Tool | Tool] | None	Tools provided to the language model to enable tool	None
response_schema	ResponseSchema | None	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	None
output_analytics	bool	Whether to output the invocation analytics. Defaults to False.	False
retry_config	RetryConfig | None	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout is used.	None
reasoning_effort	ReasoningEffort | None	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	None
web_search	bool	Whether to enable the web search. Defaults to False.	False
simplify_events	bool	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	False

Raises:

Type	Description
ValueError