Downloader
Document Processing Orchestrator Downloader Package.
Modules:
| Name | Description |
|---|---|
BaseDownloader |
Abstract base class for document downloader. |
RetryableDownloader |
Abstract base class for retryable downloader. |
DirectFileURLDownloader |
Downloader for direct file URL. |
GoogleDriveDownloader |
Downloader for Google Drive files. |
SmartCrawlDownloader |
Downloader for Smart Crawl data API. |
SmartCrawlScrapeDownloader |
Downloader for Smart Crawl scrape and general-scrape APIs. |
SmartSearchDownloader |
Downloader for Smart Search data API. |
BaseDownloader
Bases: ABC
Base class for document downloader.
download(source, output, **kwargs)
abstractmethod
Download source to the output directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The source to be downloaded. |
required |
output
|
str
|
The output directory where the downloaded source will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Returns:
| Type | Description |
|---|---|
list[str] | None
|
list[str] | None: A list of file paths of successfully downloaded files. If no files are downloaded, an empty list should be returned. Returning None is only for backward compatibility and should be avoided in new implementations. |
DirectFileURLDownloader(stream_buffer_size=65536, retry_config=None)
Bases: RetryableDownloader
A class for downloading files from a direct file URL to the defined output directory.
Initialize the DirectFileURLDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream_buffer_size
|
int
|
The size of the buffer for streaming downloads in bytes. Defaults to 64KB (65536 bytes). |
65536
|
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built by the base class with a default timeout of 30 seconds. Defaults to None. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. |
None
|
download(source, output, **kwargs)
Download source to the output directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The source to be downloaded. |
required |
output
|
str
|
The output directory where the downloaded source will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
kwargs
ca_certs_path (str, optional): The path to the CA certificates file. Defaults to None. extension (str, optional): The extension of the file to be downloaded. If not provided, the extension will be detected from the response headers or content mime type.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |
GoogleDriveDownloader(api_base_url=None, api_key=None, identifier=None, secret=None)
Bases: BaseDownloader
A class for downloading files from Google Drive using GL Connectors for Google Drive integration.
api_base_url, api_key, identifier, and secret may be supplied at construction time or per
call via download keyword arguments. In either place they must all be supplied together or all be
omitted, and they must be resolvable by download time from one of the two sources.
Initialize the GoogleDriveDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api_base_url
|
str | None
|
The base URL for the GL Connectors API. Defaults to None. |
None
|
api_key
|
str | None
|
The API key for the GL Connectors API. Defaults to None. |
None
|
identifier
|
str | None
|
The identifier for the GL Connectors user. Defaults to None. |
None
|
secret
|
str | None
|
The secret for the GL Connectors user. Defaults to None. |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If only a subset of api_base_url, api_key, identifier, and secret is provided at init. |
download(source, output, **kwargs)
Download a file from Google Drive to the output directory.
Per-call keyword arguments take precedence over init defaults.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The Google Drive file ID or URL. |
required |
output
|
str
|
The output directory where the downloaded file will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
export_format (str, optional): The export format for the file. api_base_url (str, optional): The base URL for the GL Connectors API. Override init default. api_key (str, optional): The API key for the GL Connectors API. Override init default. identifier (str, optional): The identifier for the GL Connectors user. Override init default. secret (str, optional): The secret for the GL Connectors user. Override init default.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list containing the path(s) to the successfully downloaded file(s). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If credentials or api_base_url are missing/partial, file ID cannot be extracted, user lacks Google Drive integration, or no files are returned from Google Drive. |
GoogleDriveDownloadError
|
If GL Connectors reports a failure, e.g. the file does not exist or is not accessible to the user. |
RetryableDownloader(retry_config=None)
Bases: BaseDownloader
A base downloader with built-in tenacity retry logic for transient failures.
Initialize the RetryableDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
SmartCrawlDownloader(endpoint_url, retry_config=None, api_key=None)
Bases: RetryableDownloader
A downloader for retrieving crawled records from Smart Crawl.
Initialize the SmartCrawlDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint_url
|
str
|
The URL of the Smart Crawl API endpoint. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. When a dict, it is validated into RetryConfig. When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
api_key
|
str | None
|
Value sent as the |
None
|
download(source, output, **kwargs)
Download the data from the Smart Crawl API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The smart crawl domains to be downloaded in comma separated format. |
required |
output
|
str
|
The output directory where the downloaded data will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
start_date (str): Start datetime in ISO 8601 with timezone. end_date (str): End datetime in ISO 8601 with timezone. queries (str, optional): Comma separated of search queries. schema (str, optional): Comma separated of fields to be included in the response. page (int, optional): The page number to be downloaded. page_size (int, optional): The number of items to be downloaded per page. after_timestamp (str, optional): The ISO 8601 timestamp to be used as the cursor for the next page.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |
SmartCrawlScrapeDownloader(scrape_endpoint_url, general_scrape_endpoint_url, retry_config=None, api_key=None)
Bases: RetryableDownloader
A downloader for scraping a single URL via the Smart Crawl scrape APIs.
Both endpoint URLs are required at construction. Each download must pass use_general_scraper as
True or False: True posts to /general-scrape, False posts to /scrape. The flag is never
sent in the request body.
Output JSON uses source_type html because scrape results are pre-processed HTML content.
Attributes:
| Name | Type | Description |
|---|---|---|
scrape_endpoint_url |
str
|
Domain-specific scrape API URL. |
general_scrape_endpoint_url |
str
|
General scrape API URL. |
Initialize the SmartCrawlScrapeDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scrape_endpoint_url
|
str
|
Smart Crawl domain-specific scrape API URL, e.g. |
required |
general_scrape_endpoint_url
|
str
|
Smart Crawl general scrape API URL, e.g. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. Dicts are validated into RetryConfig; None uses defaults. retry_on_exceptions is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
api_key
|
str | None
|
Value sent as the |
None
|
download(source, output, **kwargs)
Scrape a single URL via the Smart Crawl scrape APIs and save the response.
See the class docstring for how the endpoint is selected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The URL of the web page to scrape. |
required |
output
|
str
|
The output directory for the downloaded JSON file to be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
use_general_scraper (bool): When True, use general scrape. When False, use domain-specific scrape.
response_type (str, optional): "article" or "property" for general scrape. Defaults to None.
schema (dict | list[str], optional): Extraction schema for general scrape. A list of field names
is converted to {field: "string", ...}. Defaults to None.
prompt (str, optional): Prompt to guide structured extraction in general scrape.
cache (bool, optional): Whether to use cached results in general scrape. Defaults to None.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. Each file contains
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
SmartSearchDownloader(base_url, token, retry_config=None)
Bases: RetryableDownloader
A downloader for fetching web page content via the Smart Search SDK.
Initialize the SmartSearchDownloader.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base_url
|
str
|
The base URL for the Smart Search Client. |
required |
token
|
str
|
The access token for authentication. |
required |
retry_config
|
RetryConfig | dict[str, Any] | None
|
Retry configuration. - When a dict, it is validated into RetryConfig. - When None, a default RetryConfig is built. Note: retry_on_exceptions cannot be customized; it is always NETWORK_RETRY_EXCEPTIONS. Defaults to None. |
None
|
download(source, output, **kwargs)
Fetch and download a specific web page's content utilizing the Smart Search SDK.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str
|
The specific URL of the web page to fetch content from. |
required |
output
|
str
|
The output directory where the downloaded JSON data will be saved. |
required |
**kwargs
|
Any
|
Additional keyword arguments. |
{}
|
Kwargs
return_html (bool, optional): Return raw HTML if True, cleaned text if False. Defaults to False.
schema (dict | list[str], optional): Schema for custom structured data extraction. Accepts either a
full JSON Schema dict, or a list of field name strings as a shorthand (e.g. ["title", "price"]). When
a list is provided, each field is mapped to {"type": "string"} and all fields are marked as required.
Defaults to None.
Returns:
| Type | Description |
|---|---|
list[str]
|
list[str]: A list of file paths of successfully downloaded files. |