Data Pipeline Components

This module contains the base class for both “forward” and “reverse” transformer pipelines.

The primary interface that this module makes available to users is the DataPath class.

class gretel_client.transformers.data_pipeline.DataPath(*, input: str, xforms: Union[List[gretel_client.transformers.base.TransformerConfig], gretel_client.transformers.base.TransformerConfig] = None, output: str = None)

This class is a container for transformers describing each records transformations. It can also be used to transform the output field name.

It constructs the transformer instances from a list of transformer config objects and is used to process records based on the order of transformations as specified by the order of the transformer config object list.

  • input – input field name. Can be glob patterns similar to file path matching.

  • xforms – list of transformer config objects. These transformations will be applied to the data.

  • output – output field name. The output value will be written into this key.


An instance of DataPath

class gretel_client.transformers.data_pipeline.DataPipeline(data_paths: List[gretel_client.transformers.data_pipeline.DataPath])

This class is a container for data paths describing a records transformations.

It constructs a data pipeline from a list of DataPath objects and is used to process records based on the order of the data path list. You can think of it as a bundle of data paths.


data_paths – a list of data paths containing the desired input fields for processing.