Openai lm invoker

Defines a module to interact with OpenAI language models.

Authors

Henry Wicaksono (henry.wicaksono@gdplabs.id)

References

[1] https://platform.openai.com/docs/api-reference/responses

`OpenAILMInvoker(model_name, api_key=None, base_url=OPENAI_DEFAULT_URL, model_kwargs=None, default_hyperparameters=None, tools=None, response_schema=None, output_analytics=False, retry_config=None, reasoning_effort=None, reasoning_summary=None, image_generation=False, mcp_servers=None, code_interpreter=False, web_search=False, simplify_events=False)`

Bases: BaseLMInvoker

A language model invoker to interact with OpenAI language models.

This class provides support for OpenAI's Responses API schema, which is recommended by OpenAI as the preferred API to use whenever possible. Use this class unless you have a specific reason to use the Chat Completions API instead. The Chat Completions API schema is supported through the OpenAIChatCompletionsLMInvoker class.

Attributes:

Name	Type	Description
`model_id`	`str`	The model ID of the language model.
`model_provider`	`str`	The provider of the language model.
`model_name`	`str`	The name of the language model.
`client_kwargs`	`dict[str, Any]`	The keyword arguments for the OpenAI client.
`default_hyperparameters`	`dict[str, Any]`	Default hyperparameters for invoking the model.
`tools`	`list[Tool]`	The list of tools provided to the model to enable tool calling.
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary.
`output_analytics`	`bool`	Whether to output the invocation analytics.
`retry_config`	`RetryConfig`	The retry configuration for the language model.
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort.
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated.
`image_generation`	`bool`	Whether to enable image generation.
`mcp_servers`	`list[MCPServer]`	The list of MCP servers to enable MCP tool calling.
`code_interpreter`	`bool`	Whether to enable the code interpreter.
`web_search`	`bool`	Whether to enable the web search.

Basic usage

The OpenAILMInvoker can be used as follows:

lm_invoker = OpenAILMInvoker(model_name="gpt-5-nano")
result = await lm_invoker.invoke("Hi there!")

OpenAI compatible endpoints

The OpenAILMInvoker can also be used to interact with endpoints that are compatible with OpenAI's Responses API schema. This includes but are not limited to: 1. SGLang (https://github.com/sgl-project/sglang) Please note that the supported features and capabilities may vary between different endpoints and language models. Using features that are not supported by the endpoint will result in an error.

This customization can be done by setting the base_url parameter to the base URL of the endpoint:

lm_invoker = OpenAILMInvoker(
    model_name="<model-name>",
    api_key="<your-api-key>",
    base_url="<https://base-url>",
)
result = await lm_invoker.invoke("Hi there!")

Input types

The OpenAILMInvoker supports the following input types: text, document, and image. Non-text inputs can be passed as an Attachment object with the user role.

Usage example:

text = "What animal is in this image?"
image = Attachment.from_path("path/to/local/image.png")
result = await lm_invoker.invoke([text, image])

Text output

The OpenAILMInvoker generates text outputs by default. Text outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the texts (all text outputs) or text (first text output) properties.

Output example:

LMOutput(outputs=[LMOutputItem(type="text", output="Hello, there!")])

Structured output

The OpenAILMInvoker can be configured to generate structured outputs. This feature can be enabled by providing a schema to the response_schema parameter.

Structured outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the structureds (all structured outputs) or structured (first structured output) properties.

The schema must either be one of the following: 1. A Pydantic BaseModel class The structured output will be a Pydantic model. 2. A JSON schema dictionary JSON dictionary schema must be compatible with Pydantic's JSON schema, especially for complex schemas. Thus, it is recommended to create the JSON schema using Pydantic's model_json_schema method. The structured output will be a dictionary.

Usage example:

class Animal(BaseModel):
    name: str
    color: str

json_schema = Animal.model_json_schema()

lm_invoker = OpenAILMInvoker(..., response_schema=Animal)  # Using Pydantic BaseModel class
lm_invoker = OpenAILMInvoker(..., response_schema=json_schema)  # Using JSON schema dictionary

Output example:

# Using Pydantic BaseModel class outputs a Pydantic model
LMOutput(outputs=[LMOutputItem(type="structured", output=Animal(name="dog", color="white"))])

# Using JSON schema dictionary outputs a dictionary
LMOutput(outputs=[LMOutputItem(type="structured", output={"name": "dog", "color": "white"})])

When structured output is enabled, streaming is disabled.

Image generation

The OpenAILMInvoker can be configured to generate images. This feature can be enabled by setting the image_generation parameter to True.

Image outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the attachments property.

Usage example:

lm_invoker = OpenAILMInvoker(..., image_generation=True)
result = await lm_invoker.invoke("Create a picture...")
result.attachments[0].write_to_file("path/to/local/image.png")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(
            type="attachment",
            output=Attachment(filename="image.png", mime_type="image/png", data=b"..."),
        ),
    ],
)

When image generation is enabled, streaming is disabled. Image generation is only available for certain models.

Tool calling

The OpenAILMInvoker can be configured to call tools to perform certain tasks. This feature can be enabled by providing a list of Tool objects to the tools parameter.

Tool calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the tool_calls property.

Usage example:

lm_invoker = OpenAILMInvoker(..., tools=[tool_1, tool_2])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using tools..."),
        LMOutputItem(type="tool_call", output=ToolCall(id="123", name="tool_1", args={"key": "value"})),
        LMOutputItem(type="tool_call", output=ToolCall(id="456", name="tool_2", args={"key": "value"})),
    ]
)

MCP tool calling

The OpenAILMInvoker can be configured to call MCP tools to perform certain tasks. This feature can be enabled by providing a list of MCP servers to the mcp_servers parameter.

MCP calls outputs are stored in the outputs attribute of the LMOutput object and can be accessed via the mcp_calls property.

Usage example:

from gllm_inference.schema import MCPServer

mcp_server_1 = MCPServer(url="https://mcp_server_1.com", name="mcp_server_1")
lm_invoker = OpenAILMInvoker(..., mcp_servers=[mcp_server_1])

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="I'm using MCP tools..."),
        LMOutputItem(
            type="mcp_call",
            output=MCPCall(
                id="123",
                server_name="mcp_server_1",
                tool_name="mcp_tool_1",
                args={"key": "value"},
                output="The result is 10."
            ),
        ),
    ],
)

Streaming output example:

{"type": "activity", "value": {"type": "mcp_list_tools", ...}, ...}
{"type": "activity", "value": {"type": "mcp_call", ...}, ...}
{"type": "response", "value": "The result ", ...}
{"type": "response", "value": "is 10.", ...}

Note: By default, the activity token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Reasoning

The OpenAILMInvoker performs step-by-step reasoning before generating a response when reasoning models are used, such as GPT-5 models and o-series models.

The reasoning effort can be set via the reasoning_effort parameter, which guides the models on the amount of reasoning tokens to generate. Available options include minimal, low, medium, and high.

While the raw reasoning tokens are not available, the summary of the reasoning tokens can still be generated. This can be done by passing the desired summary level via the reasoning_summary parameter. Available options include auto and detailed.

Reasoning summaries are stored in the outputs attribute of the LMOutput object and can be accessed via the thinkings property.

Usage example:

lm_invoker = OpenAILMInvoker(..., reasoning_effort="high", reasoning_summary="detailed")

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="thinking", output=Reasoning(type="thinking", reasoning="I'm thinking...", ...)),
        LMOutputItem(type="text", output="Golden retriever is a good dog breed."),
    ]
)

Streaming output example:

{"type": "thinking_start", "value": "", ...}
{"type": "thinking", "value": "I'm ", ...}
{"type": "thinking", "value": "thinking...", ...}
{"type": "thinking_end", "value": "", ...}
{"type": "response", "value": "Golden retriever ", ...}
{"type": "response", "value": "is a good dog breed.", ...}

Note: By default, the thinking token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Reasoning summary is not compatible with tool calling.

Code interpreter

The OpenAILMInvoker can be configured to write and run Python code in a sandboxed environment. This is useful for solving complex problems in domains like data analysis, coding, and math. This feature can be enabled by setting the code_interpreter parameter to True.

When code interpreter is enabled, it is highly recommended to instruct the model to use the "python tool" in the system message, as "python tool" is the term recognized by the model to refer to the code interpreter.

Code execution results are stored in the outputs attribute of the LMOutput object and can be accessed via the code_exec_results property.

Usage example:

lm_invoker = OpenAILMInvoker(..., code_interpreter=True)
messages = [
    Message.system("You are a data analyst. Use the python tool to generate a file."]),
    Message.user("Show an histogram of the following data: [1, 2, 1, 4, 1, 2, 4, 2, 3, 1]"),
]
result = await lm_invoker.invoke(messages)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="text", output="The histogram is attached."),
        LMOutputItem(
            type="code_exec_result",
            output=CodeExecResult(
                id="123",
                code="import matplotlib.pyplot as plt...",
                output=[Attachment(data=b"...", mime_type="image/png")],
            ),
        ),
    ],
)

Streaming output example:

{"type": "code_start", "value": ""}', ...}
{"type": "code", "value": "import matplotlib"}', ...}
{"type": "code", "value": ".pyplot as plt..."}', ...}
{"type": "code_end", "value": ""}', ...}
{"type": "response", "value": "The histogram ", ...}
{"type": "response", "value": "is attached.", ...}

Note: By default, the code token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Web Search

The OpenAILMInvoker can be configured to search the web for relevant information. This feature can be enabled by setting the web_search parameter to True.

Web search citations are stored in the outputs attribute of the LMOutput object and can be accessed via the citations property.

Usage example:

lm_invoker = OpenAILMInvoker(..., web_search=True)

Output example:

LMOutput(
    outputs=[
        LMOutputItem(type="citation", output=Chunk(id="123", content="...", metadata={...}, score=None)),
        LMOutputItem(type="text", output="According to recent reports... ([Source](https://example.com))."),
    ],
)

Streaming output example:

{"type": "activity", "value": {"query": "search query"}, ...}
{"type": "response", "value": "According to recent ", ...}
{"type": "response", "value": "reports... ([Source](https://example.com)).", ...}

Note: By default, the activity token will be streamed with the legacy EventType.DATA event type. To use the new simplified streamed event format, set the simplify_events parameter to True during LM invoker initialization. The legacy event format support will be removed in v0.6.

Analytics tracking

The OpenAILMInvoker can be configured to output additional information about the invocation. This feature can be enabled by setting the output_analytics parameter to True.

When enabled, the following attributes will be stored in the output: 1. token_usage: The token usage. 2. duration: The duration in seconds. 3. finish_details: The details about how the generation finished.

Output example:

LMOutput(
    outputs=[...],
    token_usage=TokenUsage(input_tokens=100, output_tokens=50),
    duration=0.729,
    finish_details={"stop_reason": "end_turn"},
)

Retry and timeout

The OpenAILMInvoker supports retry and timeout configuration. By default, the max retries is set to 0 and the timeout is set to 30.0 seconds. They can be customized by providing a custom RetryConfig object to the retry_config parameter.

Retry config examples:

retry_config = RetryConfig(max_retries=0, timeout=None)  # No retry, no timeout
retry_config = RetryConfig(max_retries=5, timeout=10.0)  # 5 max retries, 10.0 seconds timeout

Usage example:

lm_invoker = OpenAILMInvoker(..., retry_config=retry_config)

Initializes a new instance of the OpenAILMInvoker class.

Parameters:

Name	Type	Description	Default
`model_name`	`str`	The name of the OpenAI model.	required
`api_key`	`str \| None`	The API key for authenticating with OpenAI. Defaults to None, in which case the `OPENAI_API_KEY` environment variable will be used. If the endpoint does not require an API key, a dummy value can be passed (e.g. "").	`None`
`base_url`	`str`	The base URL of a custom endpoint that is compatible with OpenAI's Responses API schema. Defaults to OpenAI's default URL.	`OPENAI_DEFAULT_URL`
`model_kwargs`	`dict[str, Any] \| None`	Additional model parameters. Defaults to None.	`None`
`default_hyperparameters`	`dict[str, Any] \| None`	Default hyperparameters for invoking the model. Defaults to None.	`None`
`tools`	`list[Tool \| Tool] \| None`	Tools provided to the model to enable tool calling. Defaults to None, in which case an empty list is used.	`None`
`response_schema`	`ResponseSchema \| None`	The schema of the response. If provided, the model will output a structured response as defined by the schema. Supports both Pydantic BaseModel and JSON schema dictionary. Defaults to None.	`None`
`output_analytics`	`bool`	Whether to output the invocation analytics. Defaults to False.	`False`
`retry_config`	`RetryConfig \| None`	The retry configuration for the language model. Defaults to None, in which case a default config with no retry and 30.0 seconds timeout will be used.	`None`
`reasoning_effort`	`ReasoningEffort \| None`	The reasoning effort for reasoning models. Not allowed for non-reasoning models. If None, the model will perform medium reasoning effort. Defaults to None.	`None`
`reasoning_summary`	`ReasoningSummary \| None`	The reasoning summary level for reasoning models. Not allowed for non-reasoning models. If None, no summary will be generated. Defaults to None.	`None`
`image_generation`	`bool`	Whether to enable image generation. Defaults to False.	`False`
`mcp_servers`	`list[MCPServer] \| None`	The MCP servers containing tools to be accessed by the language model. Defaults to None.	`None`
`code_interpreter`	`bool`	Whether to enable the code interpreter. Defaults to False.	`False`
`web_search`	`bool`	Whether to enable the web search. Defaults to False.	`False`
`simplify_events`	`bool`	Temporary parameter to control the streamed events format. When True, uses the simplified events format. When False, uses the legacy events format for backward compatibility. Will be removed in v0.6. Defaults to False.	`False`

Raises:

Type	Description
`ValueError`

`set_response_schema(response_schema)`

Sets the response schema for the OpenAI language model.

This method sets the response schema for the OpenAI language model. Any existing response schema will be replaced.

Parameters:

Name	Type	Description	Default
`response_schema`	`ResponseSchema \| None`	The response schema to be used.	required