Batched Data Source

A BatchedDataSourceConfig runs other data sources together with a configured order and parallelism.

BatchedDataSourceConfig — bundles other data sources into a single import.

class BatchedDataSourceConfig(run_after_import_sheet=None)

A data source that runs other data sources in a configured order.

Parameters:

run_after_import_sheet (str | None) – Optional sheet name whose import triggers this batch.

add_data_source(data_source, order, batch_data_source_type=BatchDataSourceType.NONE_PARALLEL)

Add data_source to this batch with the given execution order and BatchDataSourceType.

Return type:

BatchedDataSourceConfig

property type: DataSourceType

The DataSourceType discriminator for this configuration.

DataSourceInfo — one entry inside a BatchedDataSourceConfig.

class DataSourceInfo(data_source, order, batch_data_source_type=BatchDataSourceType.NONE_PARALLEL)

One DataSource reference inside a BatchedDataSourceConfig.

Parameters:

• data_source (DataSource) – The data source to include in the batch.

• order (int) – Execution order within the batch (lower runs first).

• batch_data_source_type (BatchDataSourceType) – Position within any surrounding parallel block.

BatchDataSourceType enum used to mark a step’s parallelism stage.

class BatchDataSourceType(*values)

Position of a batched-data-source step in a parallel block.

START_PARALLEL = 'START_PARALLEL'

First step of a parallel block.

NEXT_PARALLEL = 'NEXT_PARALLEL'

Subsequent step inside a parallel block.

END_PARALLEL = 'END_PARALLEL'

Final step of a parallel block.

NONE_PARALLEL = 'NONE_PARALLEL'

Sequential step (no parallel block).