Overview

Builder utilities for constructing GLLM Multimodal components.

This package provides factory functions for building modality converters, transformers, media toolkits, and related components from configuration.

Factory Functions

build_caption_output_formatter -- Build a caption output formatter.
build_media_toolkit -- Build a media toolkit instance for video processing.
build_media_toolkit_bulk -- Build multiple media toolkit instances in bulk.
build_modality_converter -- Build a modality converter from configuration.
build_modality_transformer -- Build a modality transformer from configuration.

Usage

from gllm_multimodal.builder import (
    build_modality_converter,
    build_modality_transformer,
    build_media_toolkit,
)

`build_caption_output_formatter(formatting_strategy)`

Build a caption output formatter based on the provided formatting strategy.

Parameters:

Name	Type	Description	Default
`formatting_strategy`	`CaptionOutputFormattingStrategy`	The formatting strategy to use.	required

Returns:

Name	Type	Description
`BaseCaptionOutputFormatter`	`BaseCaptionOutputFormatter`	The caption output formatter.

`build_media_toolkit(class_name, backend=None, processor_backends=None, **kwargs)`

Build one media toolkit component by registered class name.

This is the primary factory function for creating media toolkit components. It resolves the class name against the MediaToolkit registry and instantiates the component with the provided arguments.

For composite components (segmenters, keyframe extractors), you can additionally supply processor_backends to pre-configure per-family backend overrides without needing to call MediaToolkit.set_processor_backend manually afterwards.

Parameters:

Name	Type	Description	Default
`class_name`	`str`	Registered class name, e.g. `"AudioExtractionProcessor"` or `"FixedDurationSegmenter"`.	required
`backend`	`MediaBackend \| str \| None`	Backend selector. `None` delegates to the family's registered default backend. Pass an explicit key (e.g. `"gstreamer"` or `"ffmpeg"`) to override. Defaults to `None`.	`None`
`processor_backends`	`dict[str, str] \| None`	Per-family backend overrides applied after construction for composite components, e.g. `{"VideoClipProcessor": "ffmpeg"}`. Silently ignored for non-composites. Defaults to `None`.	`None`
`**kwargs`	`Any`	Constructor keyword arguments forwarded to the component.	`{}`

Returns:

Name	Type	Description
`MediaToolkit`	`MediaToolkit`	An instance of the requested MediaToolkit component.

Raises:

Type	Description
`ValueError`	If `class_name` or `backend` lookup fails.

Examples:

Default backend

from gllm_multimodal.builder.media_toolkit_builder import build_media_toolkit
from gllm_inference.schema import Attachment

processor = build_media_toolkit("AudioExtractionProcessor")
audio = await processor.process(Attachment.from_path("clip.mp4"))

Explicit backend + process result

processor = build_media_toolkit(
    "FixedDurationSegmenter",
    backend="gstreamer",
    segment_durations=[2.0],
)
clips = await processor.process(Attachment.from_path("video.mp4"))
# clips is a list[Attachment]

`build_media_toolkit_bulk(specs)`

Build media toolkit components from declarative specs.

This function accepts a list of specification dictionaries and builds each component using build_media_toolkit. All specs are validated against the backend registry before any component is instantiated, so errors are reported immediately.

Spec format

Each spec dictionary must have a "name" key and optionally "kwargs" and "processor_backends".

Use case: loading a list of component specs from a YAML or JSON configuration file, where all components must be validated before any is instantiated (fail-fast, all errors reported at once).

[
    {"name": "AudioExtractionProcessor", "kwargs": {"backend": "gstreamer"}},
    {"name": "FixedDurationSegmenter", "kwargs": {
        "backend": "gstreamer",
        "segment_durations": [2.0]
    }}
]

For composite components you may additionally supply "processor_backends" to pre-configure per-family backend overrides:

{
    "name": "FixedDurationSegmenter",
    "kwargs": {"backend": "gstreamer", "segment_durations": [2.0]},
    "processor_backends": {"VideoClipProcessor": "ffmpeg"}
}

Notes

name must be a registered class name (family base, concrete, or composite).
kwargs.backend is consumed by the factory and not passed as a constructor argument to leaf backend classes. For composites it is stored for nested processor resolution.
processor_backends is silently ignored for non-composite components.

Parameters:

Name	Type	Description	Default
`specs`	`list[dict[str, Any]]`	A list of component specification dictionaries.	required

Returns:

Type	Description
`list[MediaToolkit]`	list[MediaToolkit]: A list of instantiated MediaToolkit components in the
`list[MediaToolkit]`	same order as the input specs.

Raises:

Type	Description
`ValueError`	If a spec is missing `name`, a backend is unsupported, or class/backend lookup fails.

Examples:

from gllm_multimodal.builder.media_toolkit_builder import build_media_toolkit_bulk

specs = [
    {"name": "AudioExtractionProcessor", "kwargs": {"backend": "gstreamer"}},
    {"name": "FixedDurationSegmenter", "kwargs": {"segment_durations": [2.0]}},
]
components = build_media_toolkit_bulk(specs)

`build_modality_converter(source_modality, target_modality, task_type=ModalityConverterTask.AUTO, approach_type=None, preset=None, model_id=None, strategy=None, **kwargs)`

Build and initialize a modality converter instance for a given configuration.

The factory looks up the converter class based on the combination of

source_modality: input modality (e.g., Modality.IMAGE, Modality.AUDIO)
target_modality: output modality (e.g., Modality.TEXT)
task_type: conversion task (e.g., CAPTIONING, TRANSCRIPT, MERMAID, or AUTO)
approach_type: the converter's algorithmic approach; required for non-AUTO tasks, must be None for AUTO

Audio transcript approaches are separated into canonical LM_BASED, ASR, and TRANSCRIPT_FETCH values. For audio transcription, model_id selects the provider implementation and is forwarded to the LM invoker when the resolved approach is LM_BASED. Use this factory directly for audio transcription.

preset has the same meaning as for image converters: a named from_preset config.

Parameters:

Name	Type	Description	Default
`source_modality`	`Modality`	The source modality.	required
`target_modality`	`Modality`	The output modality.	required
`task_type`	`ModalityConverterTask`	The conversion task. Defaults to ModalityConverterTask.AUTO.	`AUTO`
`approach_type`	`ModalityConverterApproach \| None`	The approach for the conversion. Required for non-AUTO tasks; must be None for task_type=AUTO.	`None`
`preset`	`str \| None`	Named `from_preset` config (same semantics as image-to-text). Defaults to None.	`None`
`model_id`	`str \| ModelId \| None`	Provider-qualified audio model identifier used to infer the provider implementation when applicable. Defaults to None, which selects the default provider for the requested approach. For transcript fetching, omit this argument to select YouTube, which does not require a model name.	`None`
`strategy`	`ModalityConverterBuildStrategy \| None`	The build strategy to use. If None, the strategy is determined automatically based on the provided parameters.	`None`
`**kwargs`	`Any`	Additional keyword arguments passed to the converter, including: 1. lmrp_config (dict[str, Any]): Configuration to build an LMRP instance. Should follow the same structure as `build_lm_request_processor`. 2. Any other parameters supported by the converter's initialization method.	`{}`

Returns:

Name	Type	Description
`BaseModalityConverter`	`BaseModalityConverter`	An instance of the matching converter class.

Raises:

Type	Description
`ValueError`	If the configuration is invalid or not registered, including: - (source_modality, target_modality, task_type, approach) not registered - approach_type missing for non-AUTO task_type - approach_type provided when task_type is AUTO - Any dimension unsupported for the given combination

`build_modality_transformer(source_modality=Modality.IMAGE, target_modality=Modality.TEXT, transformer_type=ModalityTransformerType.STANDARD, router_config=None, converter_config=None, **kwargs)`

Build and initialize a modality transformer instance for a given configuration.

The factory looks up the converter class based on the combination of

source_modality: input modality (e.g., Modality.IMAGE, Modality.AUDIO)
target_modality: output modality (e.g., Modality.TEXT)
transformer_type: transformer type (e.g., ModalityTransformerType.STANDARD)

The factory then delegates construction to each class's from_config() classmethod.

Supported combinations of (source_modality, target_modality, transformer_type):

(Modality.IMAGE, Modality.TEXT, ModalityTransformerType.STANDARD): Builds a StandardImageModalityTransformer.
(Modality.IMAGE, Modality.TEXT, ModalityTransformerType.GENERIC): Builds a GenericImageModalityTransformer.

Parameters:

Name	Type	Description	Default
`source_modality`	`Modality`	The source modality. Defaults to Modality.IMAGE.	`IMAGE`
`target_modality`	`Modality`	The output modality. Defaults to Modality.TEXT.	`TEXT`
`transformer_type`	`ModalityTransformerType`	The transformer type. Defaults to ModalityTransformerType.STANDARD.	`STANDARD`
`router_config`	`RouterConfig \| None`	Additional router configuration. Defaults to None.	`None`
`converter_config`	`dict[str, ConverterConfig] \| None`	Additional converter configuration. Defaults to None.	`None`
`**kwargs`	`Any`	Additional keyword arguments passed to transformer.	`{}`

Returns:

Name	Type	Description
`BaseModalityTransformer`	`BaseModalityTransformer`	An instance of the matching transformer class.

Raises:

Type	Description
`ValueError`	If the configuration is invalid or not registered, including: - (source_modality, target_modality, transformer_type) not registered.

Examples:

Default routers and converters

from gllm_multimodal.constants import Modality, ModalityTransformerType
from gllm_multimodal.builder.modality_transformer_builder import build_modality_transformer

transformer = build_modality_transformer(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    transformer_type=ModalityTransformerType.STANDARD,
)

Custom router and default converters

from gllm_multimodal.constants import Modality, ModalityTransformerType
from gllm_multimodal.modality_transformer.schema.router_config import RouterConfig
from gllm_multimodal.builder.modality_transformer_builder import build_modality_transformer

transformer = build_modality_transformer(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    transformer_type=ModalityTransformerType.STANDARD,
    router_config=RouterConfig(
        modality=Modality.IMAGE,
        preset="multimodal",
        model_id="openai/text-embedding-3-small",
        es_url="http://localhost:9200",
        es_index_name="gllm_multimodal",
        route_mapping={"chart": "captioning", "diagram": "mermaid"},
    ),
)

Custom converters with preset

from gllm_multimodal.constants import (
    Modality, ModalityConverterTask, ModalityConverterApproach, ModalityTransformerType,
)
from gllm_multimodal.modality_transformer.schema.converter_config import ConverterConfig
from gllm_multimodal.builder.modality_transformer_builder import build_modality_transformer

transformer = build_modality_transformer(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    transformer_type=ModalityTransformerType.STANDARD,
    converter_config={
        "captioning": ConverterConfig(
            source_modality=Modality.IMAGE,
            target_modality=Modality.TEXT,
            task_type=ModalityConverterTask.CAPTIONING,
            approach_type=ModalityConverterApproach.LM_BASED,
            preset="default",
        ),
        "mermaid": ConverterConfig(
            source_modality=Modality.IMAGE,
            target_modality=Modality.TEXT,
            task_type=ModalityConverterTask.MERMAID,
            approach_type=ModalityConverterApproach.LM_BASED,
            preset="default",
        ),
    },
)

Custom converters with custom LMRP

from gllm_multimodal.constants import (
    Modality, ModalityConverterTask, ModalityConverterApproach, ModalityTransformerType,
)
from gllm_multimodal.modality_transformer.schema.converter_config import ConverterConfig
from gllm_multimodal.builder.modality_transformer_builder import build_modality_transformer

transformer = build_modality_transformer(
    source_modality=Modality.IMAGE,
    target_modality=Modality.TEXT,
    transformer_type=ModalityTransformerType.STANDARD,
    converter_config={
        "captioning": ConverterConfig(
            source_modality=Modality.IMAGE,
            target_modality=Modality.TEXT,
            task_type=ModalityConverterTask.CAPTIONING,
            approach_type=ModalityConverterApproach.LM_BASED,
            lmrp_config={
                "model_id": "google/gemini-3-flash-preview",
                "system_template": "Describe this image...",
                "user_template": "What is in this image?",
                "output_parser_type": "json",
            },
        ),
        "mermaid": ConverterConfig(
            source_modality=Modality.IMAGE,
            target_modality=Modality.TEXT,
            task_type=ModalityConverterTask.MERMAID,
            approach_type=ModalityConverterApproach.LM_BASED,
            lmrp_config={
                "model_id": "openai/gpt-5.2-latest",
                "system_template": "Generate mermaid...",
                "user_template": "Convert to mermaid...",
            },
        ),
    },
)