Data Pipeline Components¶
This module contains the base class for both “forward” and “reverse” transformer pipelines.
The primary interface that this module makes available to users is the
DataPath(*, input: str, xforms: Union[List[gretel_client.transformers.base.TransformerConfig], gretel_client.transformers.base.TransformerConfig] = None, output: str = None)¶
This class is a container for transformers describing each records transformations. It can also be used to transform the output field name.
It constructs the transformer instances from a list of transformer config objects and is used to process records based on the order of transformations as specified by the order of the transformer config object list.
input – input field name. Can be glob patterns similar to file path matching.
xforms – list of transformer config objects. These transformations will be applied to the data.
output – output field name. The output value will be written into this key.
An instance of
This class is a container for data paths describing a records transformations.
It constructs a data pipeline from a list of
DataPathobjects and is used to process records based on the order of the data path list. You can think of it as a bundle of data paths.
data_paths – a list of data paths containing the desired input fields for processing.