Downloader
Document Processing Orchestrator Downloader Package.
Modules:
| Name | Description |
|---|---|
BaseDownloader |
Abstract base class for document downloader. |
RetryableDownloader |
Abstract base class for retryable downloader. |
DirectFileURLDownloader |
Downloader for direct file URL. |
GoogleDriveDownloader |
Downloader for Google Drive files. |
SmartCrawlDownloader |
Downloader for Smart Crawl data API. |
SmartCrawlScrapeDownloader |
Downloader for Smart Crawl scrape API. |
SmartSearchDownloader |
Downloader for Smart Search data API. |
BaseDownloader
Bases: ABC
Base class for document downloader.
download(source, output, **kwargs)
abstractmethod
Download source to the output directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The source to be downloaded. |
required |
output
|
str
|
The output directory where the downloaded source will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
list[str] | None
|
list[str] | None: A list of file paths of successfully downloaded files. If no files are downloaded, an empty list should be returned. Returning None is only for backward compatibility and should be avoided in new implementations. |
DirectFileURLDownloader(stream_buffer_size=65536, retry_config=None)
Bases: RetryableDownloader
A class for downloading files from a direct file URL to the defined output directory.
Initialize the DirectFileURLDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream_buffer_size
|
int
|
The size of the buffer for streaming downloads in bytes. Defaults to 64KB (65536 bytes). |
65536
|
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built by the base class with a default timeout of 30 seconds. Defaults to None. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. |
None
|
download(source, output, **kwargs)
Download source to the output directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The source to be downloaded. |
required |
output
|
str
|
The output directory where the downloaded source will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
kwargs
ca_certs_path (str, optional): The path to the CA certificates file. Defaults to None. extension (str, optional): The extension of the file to be downloaded. If not provided, the extension will be detected from the response headers or content mime type.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |
GoogleDriveDownloader(api_base_url=None, api_key=None, identifier=None, secret=None)
Bases: BaseDownloader
A class for downloading files from Google Drive using GL Connectors for Google Drive integration.
api_base_url, api_key, identifier, and secret may be supplied at construction time or per
call via download keyword arguments. In either place they must all be supplied together or all be
omitted, and they must be resolvable by download time from one of the two sources.
Initialize the GoogleDriveDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api_base_url
|
str | None
|
The base URL for the GL Connectors API. Defaults to None. |
None
|
api_key
|
str | None
|
The API key for the GL Connectors API. Defaults to None. |
None
|
identifier
|
str | None
|
The identifier for the GL Connectors user. Defaults to None. |
None
|
secret
|
str | None
|
The secret for the GL Connectors user. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If only a subset of api_base_url, api_key, identifier, and secret is provided at init. |
download(source, output, **kwargs)
Download a file from Google Drive to the output directory.
Per-call keyword arguments take precedence over init defaults.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The Google Drive file ID or URL. |
required |
output
|
str
|
The output directory where the downloaded file will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
export_format (str, optional): The export format for the file. api_base_url (str, optional): The base URL for the GL Connectors API. Override init default. api_key (str, optional): The API key for the GL Connectors API. Override init default. identifier (str, optional): The identifier for the GL Connectors user. Override init default. secret (str, optional): The secret for the GL Connectors user. Override init default.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list containing the path(s) to the successfully downloaded file(s). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If credentials or api_base_url are missing/partial, file ID cannot be extracted, user lacks Google Drive integration, or no files are returned from Google Drive. |
GoogleDriveDownloadError
|
If GL Connectors reports a failure, e.g. the file does not exist or is not accessible to the user. |
RetryableDownloader(retry_config=None)
Bases: BaseDownloader
A base downloader with built-in tenacity retry logic for transient failures.
Initialize the RetryableDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
SmartCrawlDownloader(endpoint_url, retry_config=None)
Bases: RetryableDownloader
A downloader for retrieving crawled records from Smart Crawl.
Initialize the SmartCrawlDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint_url
|
str
|
The URL of the Smart Crawl API endpoint. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
download(source, output, **kwargs)
Download the data from the Smart Crawl API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The smart crawl domains to be downloaded in comma separated format. |
required |
output
|
str
|
The output directory where the downloaded data will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
start_date (str): Start datetime in ISO 8601 with timezone. end_date (str): End datetime in ISO 8601 with timezone. queries (str, optional): Comma separated of search queries. schema (str, optional): Comma separated of fields to be included in the response. page (int, optional): The page number to be downloaded. page_size (int, optional): The number of items to be downloaded per page. after_timestamp (str, optional): The ISO 8601 timestamp to be used as the cursor for the next page.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |
SmartCrawlScrapeDownloader(endpoint_url, retry_config=None)
Bases: RetryableDownloader
A downloader for scraping a single URL via the Smart Crawl scrape API.
Initialize the SmartCrawlScrapeDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint_url
|
str
|
The URL of the Smart Crawl scrape API endpoint. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
download(source, output, **kwargs)
Scrape a single URL via the Smart Crawl Scrape API and save the response.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The specific URL of the web page to scrape. |
required |
output
|
str
|
The output directory where the downloaded JSON data will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
response_type (str, optional): Response type for general scrape mode, either "article" or "property".
When a schema is set, the API defaults to "article" if not provided. Defaults to None.
schema (dict | list[str], optional): Custom extraction schema for general scrape mode. Accepts either a
full schema dict (e.g. {"title": "string"}), or a list of field name strings as a shorthand
(e.g. ["title", "content"]). When a list is provided, each field is mapped to string.
Defaults to None.
prompt (str, optional): Prompt for general scrape mode to guide structured extraction.
use_general_scraper (bool, optional): Whether to bypass domain-specific scrapers and use general scrape
mode. Required for general extraction on mapped domains (e.g. OLX news URLs). Unmapped domains still
fall back to general scrape when this is False. The API defaults to False if not provided.
Defaults to None.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |
SmartSearchDownloader(base_url, token, retry_config=None)
Bases: RetryableDownloader
A downloader for fetching web page content via the Smart Search SDK.
Initialize the SmartSearchDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_url
|
str
|
The base URL for the Smart Search Client. |
required |
token
|
str
|
The access token for authentication. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. - When a dict, it is validated into RetryConfig. - When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
download(source, output, **kwargs)
Fetch and download a specific web page's content utilizing the Smart Search SDK.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The specific URL of the web page to fetch content from. |
required |
output
|
str
|
The output directory where the downloaded JSON data will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
return_html (bool, optional): Return raw HTML if True, cleaned text if False. Defaults to False.
schema (dict | list[str], optional): Schema for custom structured data extraction. Accepts either a
full JSON Schema dict, or a list of field name strings as a shorthand (e.g. ["title", "price"]). When
a list is provided, each field is mapped to {"type": "string"} and all fields are marked as required.
Defaults to None.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |