Overview

Detectors consumed by media toolkit components.

This module provides detectors that locate content, such as on-screen text, in images and decoded video frames.

Exported Classes

BaseTextDetector -- Abstract base for text-mask detectors.
PPOCRTextDetector -- Lazy PP-OCRv6 ONNX text detector.

`BaseTextDetector(min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)`

Bases: MediaToolkit[Attachment, list[Attachment]], ABC

Define the text detector role for media toolkit components.

Detectors expose two public entry points:

process accepts an image attachment and returns one cropped PNG attachment per detected text region.
detect runs on a decoded BGR uint8 array and returns either a boolean text mask or a list of (x, y, width, height) text boxes. Consumers that already hold decoded frames, such as keyframe extractors, call it directly to avoid re-encoding each frame, and pass the result through resolve_text_mask when they need a mask.

Segmentation-style detectors return a mask; box-style detectors, such as object detectors, return boxes. Subclasses implement only detect and forward min_region_area and region_padding to super().__init__. Construction must be synchronous and free of I/O so detectors can be built from declarative specs. Any slow preparation, such as downloading model weights, belongs in ensure_ready, which consumers await before calling detect. Detectors holding native resources release them in close.

Usage Example

Crop text regions (process):

from gllm_inference.schema import Attachment
from gllm_multimodal.media_toolkit.media_toolkit import MediaToolkit

detector = MediaToolkit.build("PPOCRTextDetector")

image = Attachment.from_path("/path/to/slide.png")
crops = await detector.process(image)
for crop in crops:
    print(crop.filename, crop.metadata["bbox"])  # slide_text_0.png [x, y, width, height]

Text mask (detect):

import numpy as np
from PIL import Image
from gllm_multimodal.media_toolkit.media_toolkit import MediaToolkit

detector = MediaToolkit.build("PPOCRTextDetector")
await detector.ensure_ready()

frame = np.asarray(Image.open("/path/to/slide.png").convert("RGB"))[:, :, ::-1]  # RGB -> BGR
mask = detector.detect(frame)  # bool array of shape (height, width)

Box detector:

import numpy as np
from gllm_multimodal.media_toolkit.detector.text_detector import BaseTextDetector
from gllm_multimodal.media_toolkit.detector.text_detector.base_text_detector import TextBox


class MyBoxTextDetector(BaseTextDetector):
    def detect(self, frame: np.ndarray) -> list[TextBox]:
        return [(12, 40, 200, 32)]  # (x, y, width, height)

Attributes:

Name	Type	Description
`min_region_area`	`int`	Minimum text area in pixels a region needs to be returned by `process`: foreground pixels for a mask region, `width * height` for a box. Smaller regions are treated as noise.
`region_padding`	`int`	Pixels added around each region's bounding box before cropping.

Validate the region options used by process.

Parameters:

Name	Type	Description	Default
`min_region_area`	`int`	Minimum text area in pixels a region needs to be returned by `process`. Must be at least 1. Defaults to 16.	`DEFAULT_MIN_REGION_AREA`
`region_padding`	`int`	Pixels added around each region's bounding box before cropping. Must be non-negative. Defaults to 2.	`DEFAULT_REGION_PADDING`

Raises:

Type	Description
`ValueError`	If `min_region_area` or `region_padding` is out of range.

`close()`

Release resources held by the detector.

The default implementation is a no-op for detectors that hold no native resources. Implementations must be idempotent.

`detect(frame)` `abstractmethod`

Detect text in a BGR analysis frame.

Parameters:

Name	Type	Description	Default
`frame`	`ndarray`	Three-channel BGR `uint8` analysis frame.	required

Returns:

Name	Type	Description
`TextDetection`	`TextDetection`	Either a two-dimensional boolean mask matching `frame` height and width, or a list of integer `(x, y, width, height)` boxes that lie inside `frame`, with positive width and height.

`ensure_ready()` `async`

Prepare the detector for inference.

The default implementation is a no-op for detectors that need no preparation. Implementations must be idempotent.

`resolve_input_size(width, height)`

Return the (width, height) analysis frames should be resized to.

Consumers resize each frame to this size before calling detect so the grayscale frame, the mask, and inference all share one resolution.

Parameters:

Name	Type	Description	Default
`width`	`int`	Source frame width in pixels.	required
`height`	`int`	Source frame height in pixels.	required

Returns:

Type	Description
`tuple[int, int] \| None`	tuple[int, int] \| None: Preferred analysis size, or `None` when the detector accepts any size and the consumer's own bounds apply.

`PPOCRTextDetector(source=DEFAULT_MODEL_SOURCE, cache_dir=None, limit_side_len=DEFAULT_LIMIT_SIDE_LEN, binary_threshold=DEFAULT_BINARY_THRESHOLD, num_threads=None, min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)`

Bases: BaseTextDetector

Run a pinned PP-OCRv6 model as a synchronous text-mask detector.

Construction only validates options. Built-in weights are downloaded by ensure_ready and the ONNX Runtime session is created lazily, under a lock, on the first detect call. Both are released by close. process awaits ensure_ready itself, so callers only need to await it before calling detect directly.

Requires the video-ffmpeg or video-gst extra, which installs ONNX Runtime.

Usage Example

Crop text regions (process):

from gllm_inference.schema import Attachment
from gllm_multimodal.media_toolkit.detector.text_detector import PPOCRTextDetector

with PPOCRTextDetector(source="ppocrv6_small", cache_dir="~/.cache/gllm/ppocr") as detector:
    crops = await detector.process(Attachment.from_path("/path/to/slide.png"))

Text mask (detect):

import numpy as np
from PIL import Image
from gllm_multimodal.media_toolkit.detector.text_detector import PPOCRTextDetector

with PPOCRTextDetector(binary_threshold=0.3) as detector:
    await detector.ensure_ready()
    frame = np.asarray(Image.open("/path/to/slide.png").convert("RGB"))[:, :, ::-1]  # RGB -> BGR
    mask = detector.detect(frame)  # bool array of shape (height, width)

Local ONNX weights:

from gllm_multimodal.media_toolkit.detector.text_detector import PPOCRTextDetector

# Local paths skip the download; cache_dir must stay None.
detector = PPOCRTextDetector(source="/models/ppocr_det.onnx", num_threads=4)

Preprocessing follows PaddleOCR's detection pipeline: 1. Frames stay in BGR order, matching DecodeImage(img_mode=BGR). 2. The longer side is bounded by limit_side_len while preserving the aspect ratio, then each side is rounded to a multiple of 32, matching DetResizeForTest with limit_type="max". 3. Pixels are scaled to [0, 1] and normalized with ImageNet mean and std in BGR channel order, matching NormalizeImage.

Attributes:

Name	Type	Description
`source`	`str \| Path`	Built-in model name (`str`) or readable local ONNX path.
`cache_dir`	`str \| Path \| None`	Persistent weight cache root for built-in models.
`limit_side_len`	`int`	Maximum longer side of the model input in pixels.
`binary_threshold`	`float`	Probability cutoff above which a pixel is text.
`num_threads`	`int \| None`	Optional CPU intra-op thread count.
`min_region_area`	`int`	Minimum number of text pixels a region needs to be returned by `process`.
`region_padding`	`int`	Pixels added around each region's bounding box before cropping.

Validate the model source and inference options without I/O.

Parameters:

Name	Type	Description	Default
`source`	`str \| Path`	Exact built-in model name given as a `str`, one of `ppocrv6_tiny`, `ppocrv6_small`, or `ppocrv6_medium`, or a path to an existing readable local ONNX file. A `str` built-in name takes priority over a same-named local file; a `Path` is always treated as a local file, so pass a `Path` to load such a file. Defaults to `"ppocrv6_small"`.	`DEFAULT_MODEL_SOURCE`
`cache_dir`	`str \| Path \| None`	Persistent cache root for built-in weights. When `None`, verified weights are kept in memory for the detector lifetime. Must be `None` for a local path source. Defaults to None.	`None`
`limit_side_len`	`int`	Maximum longer side of the model input in pixels. Larger frames are downscaled with their aspect ratio preserved; smaller frames are only rounded to multiples of 32. Must be at least 32. Defaults to 960.	`DEFAULT_LIMIT_SIDE_LEN`
`binary_threshold`	`float`	Cutoff applied to the model's per-pixel text probability map. Pixels whose probability is strictly greater than this value are marked as text. Lower values keep faint or small text at the cost of more false positives; higher values keep only confident text. The default matches PaddleOCR's `DBPostProcess.thresh`. Defaults to 0.2.	`DEFAULT_BINARY_THRESHOLD`
`num_threads`	`int \| None`	ONNX Runtime CPU intra-op thread count. `None` or `0` lets ONNX Runtime choose. Defaults to None.	`None`
`min_region_area`	`int`	Minimum number of text pixels a connected region needs to be returned by `process`. Must be at least 1. Defaults to 16.	`DEFAULT_MIN_REGION_AREA`
`region_padding`	`int`	Pixels added around each region's bounding box before cropping in `process`. Must be non-negative. Defaults to 2.	`DEFAULT_REGION_PADDING`

Raises:

Type	Description
`ValueError`	If the source is unsupported, `cache_dir` is supplied for a local path, or an inference option is invalid.
`FileNotFoundError`	If a `Path` source, or an existing string path, is not a readable file.

`enter()`

Return this detector for synchronous context management.

Returns:

Name	Type	Description
`PPOCRTextDetector`	`PPOCRTextDetector`	This open detector instance.

Raises:

Type	Description
`RuntimeError`	If the detector has already been closed.

`exit(exc_type, exc_value, traceback)`

Release the owned ONNX session and ephemeral model bytes.

Parameters:

Name	Type	Description	Default
`exc_type`	`object`	Exception type supplied by the context manager.	required
`exc_value`	`object`	Exception value supplied by the context manager.	required
`traceback`	`object`	Traceback supplied by the context manager.	required

`close()`

Idempotently release session and memory-backed model bytes.

This method is safe to call multiple times and from multiple threads.

`detect(frame)`

Detect text in one BGR uint8 frame.

Parameters:

Name	Type	Description	Default
`frame`	`ndarray`	Three-channel BGR analysis frame.	required

Returns:

Type	Description
`ndarray`	np.ndarray: Two-dimensional boolean mask with the input frame's height and width.

Raises:

Type	Description
`RuntimeError`	If the detector is closed or `ensure_ready` was not awaited.
`ValueError`	If the frame or ONNX graph output is malformed.

`ensure_ready()` `async`

Resolve built-in weights once so detect can create a session.

Local path sources need no preparation. Concurrent callers share one download.

Raises:

Type	Description
`RuntimeError`	If the detector has been closed.
`OSError`	If downloading or caching built-in weights fails.

`resolve_input_size(width, height)`

Return the aspect-preserving PP-OCR input size for a frame.

Parameters:

Name	Type	Description	Default
`width`	`int`	Source frame width in pixels.	required
`height`	`int`	Source frame height in pixels.	required

Returns:

Type	Description
`tuple[int, int]`	tuple[int, int]: `(width, height)` bounded by `limit_side_len` with both sides rounded to positive multiples of 32.

Raises:

Type	Description
`ValueError`	If `width` or `height` is not positive.

Overview

Exported Classes

BaseTextDetector(min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)

close()

detect(frame) abstractmethod

ensure_ready() async

resolve_input_size(width, height)

PPOCRTextDetector(source=DEFAULT_MODEL_SOURCE, cache_dir=None, limit_side_len=DEFAULT_LIMIT_SIDE_LEN, binary_threshold=DEFAULT_BINARY_THRESHOLD, num_threads=None, min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)

__enter__()

__exit__(exc_type, exc_value, traceback)

close()

detect(frame)

ensure_ready() async

resolve_input_size(width, height)

`BaseTextDetector(min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)`

`close()`

`detect(frame)` `abstractmethod`

`ensure_ready()` `async`

`resolve_input_size(width, height)`

`PPOCRTextDetector(source=DEFAULT_MODEL_SOURCE, cache_dir=None, limit_side_len=DEFAULT_LIMIT_SIDE_LEN, binary_threshold=DEFAULT_BINARY_THRESHOLD, num_threads=None, min_region_area=DEFAULT_MIN_REGION_AREA, region_padding=DEFAULT_REGION_PADDING)`

`enter()`

`exit(exc_type, exc_value, traceback)`

`close()`

`detect(frame)`

`ensure_ready()` `async`

`resolve_input_size(width, height)`