Storage
Storage module for gllm_agents.
This module provides comprehensive storage functionality including object storage clients, storage providers for tool outputs, and configuration management.
BaseObjectStorageClient
Bases: ABC
Abstract base class for object storage clients.
delete(object_key)
abstractmethod
Delete data from object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object in the storage |
required |
generate_presigned_url(object_key, expires=24, response_headers=None)
abstractmethod
Generate a presigned URL for accessing the object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object in the storage |
required |
expires |
int
|
The number of hours the URL is valid for |
24
|
response_headers |
dict[str, str] | None
|
Additional headers to include in the response |
None
|
Returns:
| Type | Description |
|---|---|
str
|
The presigned URL |
get(object_key)
abstractmethod
Get data from object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object in the storage |
required |
Returns:
| Type | Description |
|---|---|
bytes
|
The binary data of the object |
object_exists(object_key)
abstractmethod
Check if an object exists in the storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object to check. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the object exists, False otherwise. |
upload(object_key, file_stream, filename=None, content_type=None, metadata=None)
abstractmethod
Upload data to object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object in the storage |
required |
file_stream |
bytes | BinaryIO
|
The binary data to upload |
required |
filename |
str | None
|
The name of the file |
None
|
content_type |
str | None
|
The content type of the file |
None
|
metadata |
dict[str, str] | None
|
Additional metadata to store with the object |
None
|
Returns:
| Type | Description |
|---|---|
str
|
The key of the uploaded object |
BaseStorageProvider
Bases: ABC
Base interface for storage providers.
This abstract class defines the contract that all storage providers must implement to store and retrieve tool outputs.
clear()
abstractmethod
Clear all stored data.
Warning
This operation is irreversible
delete(key)
abstractmethod
Delete data by key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
Note
Should not raise error if key doesn't exist
exists(key)
abstractmethod
Check if key exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier to check |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if key exists, False otherwise |
get_presigned_url(key, expires_hours=24)
Generate presigned URL for direct access (optional).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Storage key |
required |
expires_hours |
int
|
URL expiration in hours |
24
|
Returns:
| Type | Description |
|---|---|
str | None
|
Presigned URL if supported, None otherwise |
list_keys(prefix='')
abstractmethod
List all keys with optional prefix filter.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prefix |
str
|
Optional prefix to filter keys |
''
|
Returns:
| Type | Description |
|---|---|
list[str]
|
List of matching keys |
retrieve(key)
abstractmethod
Retrieve data by key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The stored data |
Raises:
| Type | Description |
|---|---|
KeyError
|
If key not found |
StorageError
|
If retrieval operation fails |
store(key, data)
abstractmethod
Store data with the given key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
data |
Any
|
Data to store (must be serializable) |
required |
Raises:
| Type | Description |
|---|---|
StorageError
|
If storage operation fails |
InMemoryStorageProvider()
Bases: BaseStorageProvider
In-memory storage provider for fast access to small data.
This provider stores all data in memory, providing the fastest access times but limited by available RAM.
Best for: - Small to medium datasets - Temporary storage - Development and testing - High-frequency access patterns
Initialize in-memory storage.
count: int
property
Get number of stored items.
Returns:
| Type | Description |
|---|---|
int
|
Number of items in storage |
size_bytes: int
property
Get approximate memory usage in bytes.
Returns:
| Type | Description |
|---|---|
int
|
Approximate memory usage in bytes |
clear()
Clear all data from memory.
delete(key)
Delete data from memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
exists(key)
Check if key exists in memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier to check |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if key exists, False otherwise |
list_keys(prefix='')
List all keys with optional prefix.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prefix |
str
|
Optional prefix to filter keys |
''
|
Returns:
| Type | Description |
|---|---|
list[str]
|
List of matching keys |
retrieve(key)
Retrieve data from memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The stored data |
Raises:
| Type | Description |
|---|---|
KeyError
|
If key not found |
store(key, data)
Store data in memory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
data |
Any
|
Data to store |
required |
Raises:
| Type | Description |
|---|---|
StorageError
|
If storage operation fails |
MinioConfig(endpoint, access_key, secret_key, bucket, secure=True)
dataclass
Configuration for MinIO object storage client.
Attributes:
| Name | Type | Description |
|---|---|---|
endpoint |
str
|
MinIO server endpoint URL |
access_key |
str
|
Access key for authentication |
secret_key |
str
|
Secret key for authentication |
bucket |
str
|
Bucket name to use for storage |
secure |
bool
|
Whether to use HTTPS (defaults to True) |
from_env()
classmethod
Create MinioConfig from environment variables.
Expected environment variables: - OBJECT_STORAGE_URL - OBJECT_STORAGE_USER - OBJECT_STORAGE_PASSWORD - OBJECT_STORAGE_BUCKET - OBJECT_STORAGE_SECURE (optional, defaults to True)
Returns:
| Type | Description |
|---|---|
MinioConfig
|
MinioConfig instance |
Raises:
| Type | Description |
|---|---|
ValueError
|
If required environment variables are not set |
MinioObjectStorage(config=None, ensure_bucket=True)
Bases: BaseObjectStorageClient
Implementation of ObjectStorageInterface using Minio.
Initialize MinioObjectStorage with configuration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config |
MinioConfig | None
|
MinioConfig instance. If None, will attempt to load from environment variables. |
None
|
ensure_bucket |
bool
|
Whether to ensure bucket exists during initialization (optional). Defaults to True. |
True
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If configuration is invalid or incomplete |
delete(object_key)
Delete data from Minio object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object to delete |
required |
Raises:
| Type | Description |
|---|---|
ConnectionError
|
If there's a network or connection issue |
generate_presigned_url(object_key, expires=24, response_headers=None)
Generate a presigned URL for accessing the object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object |
required |
expires |
int
|
Expiration time in hours (defaults to 24) |
24
|
response_headers |
dict[str, str] | None
|
Additional response headers (optional) |
None
|
Returns:
| Type | Description |
|---|---|
str
|
A presigned URL |
Raises:
| Type | Description |
|---|---|
ValueError
|
If expiration time is not positive |
ConnectionError
|
If there's an issue generating the URL |
get(object_key)
Get data from Minio object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object to retrieve |
required |
Returns:
| Type | Description |
|---|---|
bytes
|
The object data as bytes |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the object is not found |
ConnectionError
|
If there's a network or connection issue |
list_objects(prefix='')
List objects in the bucket with optional prefix filter.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prefix |
str
|
Optional prefix to filter objects |
''
|
Returns:
| Type | Description |
|---|---|
list[str]
|
List of object keys |
Raises:
| Type | Description |
|---|---|
ConnectionError
|
If there's an issue listing objects |
object_exists(object_key)
Check if an object exists in the MinIO bucket.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key of the object to check. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if the object exists, False otherwise. |
Raises:
| Type | Description |
|---|---|
ConnectionError
|
If there's a network or connection issue (excluding not found errors) |
upload(object_key, file_stream, filename=None, content_type=None, metadata=None)
Upload data to Minio object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
object_key |
str
|
The key to store the object under |
required |
file_stream |
bytes | BinaryIO
|
The data to upload (bytes or file-like object) |
required |
filename |
str | None
|
The filename of the data (optional) |
None
|
content_type |
str | None
|
The content type of the data (optional) |
None
|
metadata |
dict[str, str] | None
|
Additional metadata to store with the object (optional) |
None
|
Returns:
| Type | Description |
|---|---|
str
|
The object key of the uploaded data |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file stream is empty or invalid |
ConnectionError
|
If there's an issue connecting to Minio |
Exception
|
For other unexpected errors during upload |
ObjectStorageProvider(client, prefix='', use_json=False)
Bases: BaseStorageProvider
Object storage provider for S3-compatible storage.
Works with any S3-compatible storage including AWS S3, MinIO, Google Cloud Storage (with S3 compatibility), etc.
Best for: - Very large datasets - Distributed systems - Cloud deployments - Long-term storage - Multi-region access
Initialize object storage provider.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
client |
BaseObjectStorageClient
|
Object storage client instance |
required |
prefix |
str
|
Prefix for all keys (like a directory) |
''
|
use_json |
bool
|
Use JSON format (True) or pickle (False) |
False
|
clear()
Clear all objects with the configured prefix.
Warning
This is a dangerous operation!
delete(key)
Delete object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
exists(key)
Check if object exists.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier to check |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True if key exists, False otherwise |
get_presigned_url(key, expires_hours=24)
Generate presigned URL for direct access.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Storage key |
required |
expires_hours |
int
|
URL expiration in hours |
24
|
Returns:
| Type | Description |
|---|---|
str | None
|
Presigned URL for direct access |
Raises:
| Type | Description |
|---|---|
StorageError
|
If URL generation fails |
list_keys(prefix='')
List all keys with optional prefix.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prefix |
str
|
Optional prefix to filter keys |
''
|
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
list_keys is not implemented |
retrieve(key)
Retrieve data from object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The stored data |
Raises:
| Type | Description |
|---|---|
KeyError
|
If key not found |
StorageError
|
If retrieval operation fails |
store(key, data)
Store data in object storage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str
|
Unique identifier for the data |
required |
data |
Any
|
Data to store |
required |
Raises:
| Type | Description |
|---|---|
StorageError
|
If storage operation fails |
StorageConfig(storage_type=StorageType.MEMORY, object_prefix=OBJECT_STORAGE_PREFIX, object_use_json=False)
dataclass
Configuration for storage providers.
from_env()
classmethod
Create StorageConfig from environment variables.
StorageError
Bases: Exception
Base exception for storage operations.
StorageProviderFactory
Factory for creating storage providers based on configuration.
create(config, object_storage_client=None)
staticmethod
Create storage provider based on configuration.
create_from_env(object_storage_client=None)
staticmethod
Create storage provider from environment variables.
StorageType
Bases: StrEnum
Supported storage types.