Data Pipeline Components¶
This module contains the base class for both “forward” and “reverse” transformer pipelines.
The primary interface that this module makes available to users is the DataPath class.
-
class
gretel_client.transformers.data_pipeline.DataPath(*, input: str, xforms: Union[List[gretel_client.transformers.base.TransformerConfig], gretel_client.transformers.base.TransformerConfig] = None, output: str = None)¶ This class is a container for transformers describing each records transformations. It can also be used to transform the output field name.
It constructs the transformer instances from a list of transformer config objects and is used to process records based on the order of transformations as specified by the order of the transformer config object list.
- Parameters
input – input field name. Can be glob patterns similar to file path matching.
xforms – list of transformer config objects. These transformations will be applied to the data.
output – output field name. The output value will be written into this key.
- Returns
An instance of
DataPath
-
class
gretel_client.transformers.data_pipeline.DataPipeline(data_paths: List[gretel_client.transformers.data_pipeline.DataPath])¶ This class is a container for data paths describing a records transformations.
It constructs a data pipeline from a list of
DataPathobjects and is used to process records based on the order of the data path list. You can think of it as a bundle of data paths.- Parameters
data_paths – a list of data paths containing the desired input fields for processing.