Skip to content

Modality Converter

Abstract base class for modality conversion in Gen AI applications.

This module defines the common interface for converting between different media modalities (image, audio, video) and text. All modality converters in the GLLM ecosystem extend this class.

BaseModalityConverter()

Bases: ABC

Abstract base class for modality conversion in Gen AI applications.

This class defines the common interface for converting between different media modalities (image, audio, video) and text. Subclasses implement specific conversion strategies (e.g., OCR, captioning, speech-to-text).

The converter follows a simple contract:

  1. Accept a source (file path, URL, or raw bytes).
  2. Validate and process the source.
  3. Return a TextResult containing the converted text and optional metadata.

Subclasses

Usage

The recommended way to construct any converter is through the builder:

from gllm_multimodal.builder.modality_converter_builder import build_modality_converter
from gllm_multimodal.constants import Modality, ModalityConverterTask, ModalityConverterApproach

converter = build_modality_converter(
    source_modality=Modality.AUDIO,
    target_modality=Modality.TEXT,
    task_type=ModalityConverterTask.TRANSCRIPT,
    approach_type=ModalityConverterApproach.LM_BASED,
    lmrp_config={"model_id": "google/gemini-2.5-flash"},
)
# result is a TextResult with .text and .metadata
result = await converter.convert_to_text_result(source="path/to/audio.mp3")
print(result.text)

For contributors adding a new converter subclass, implement convert:

class MyConverter(BaseModalityConverter):
    async def convert(self, source: str | bytes, **kwargs) -> TextResult:
        # implementation here
        ...

Attributes:

Name Type Description
_logger

Logger instance scoped to the concrete class name.

Initialize the base modality converter with logging capabilities.

convert(source, **kwargs) abstractmethod async

Execute the modality conversion process.

Subclasses must implement this method to define the conversion logic. The method receives a validated source and returns a structured result.

Parameters:

Name Type Description Default
source str | bytes

The source of the modality to convert. Can be a file path, URL string, or raw bytes.

 required
**kwargs Any

Additional conversion parameters. Implementations may accept keyword arguments such as language, prompt, or provider-specific options.

 {}

Returns:

Name Type Description
TextResult TextResult

The result of the conversion, containing the extracted text and
TextResult

optional metadata.

Raises:

Type Description
ValueError

If source is an empty string or missing.
TypeError

If source is not a string or bytes.
NotImplementedError

If the subclass does not override this method.