Skip to content

cdm_data_loaders.core.fields

Common fields and defaults for data loading and wrangling pipelines.

PIPELINE_DIR_NAME module-attribute 

PIPELINE_DIR_NAME: Final[str] = 'pipeline_metadata'

Directory where dlt will put run metadata, schemas, etc.

REPORT_FILE_NAME module-attribute 

REPORT_FILE_NAME: Final[str] = 'report.txt'

Name of the report file.

LoaderFileFormatEnum

Bases: StrEnum

Valid values for loader file format.

Source code in src/cdm_data_loaders/core/fields.py 
51
52
53
54
55
56
class LoaderFileFormatEnum(StrEnum):
    """Valid values for loader file format."""

    ICEBERG = ICEBERG
    JSONL = JSONL
    PARQUET = PARQUET

expand_cts_glob_wildcards 

expand_cts_glob_wildcards(value: str) -> str

Replace CTS wildcard tokens in value with glob metacharacters.

Parameters:

Name Type Description Default
value str

raw or CTS-tokenized glob pattern

 required

Returns:

Type Description
str

glob pattern with CTS wildcard tokens expanded
Source code in src/cdm_data_loaders/core/fields.py 
143
144
145
146
147
148
149
150
151
def expand_cts_glob_wildcards(value: str) -> str:
    """Replace CTS wildcard tokens in ``value`` with glob metacharacters.

    :param value: raw or CTS-tokenized glob pattern
    :type value: str
    :return: glob pattern with CTS wildcard tokens expanded
    :rtype: str
    """
    return CTS_GLOB_WILDCARD_PATTERN.sub(_replace_cts_glob_wildcard, value)

file_glob_description 

file_glob_description(summary: str) -> str

Build a CLI field description for file_glob including CTS wildcard tokens.

Parameters:

Name Type Description Default
summary str

short description of matched input files

 required

Returns:

Type Description
str

field description with CTS wildcard token reference
Source code in src/cdm_data_loaders/core/fields.py 
154
155
156
157
158
159
160
161
162
163
def file_glob_description(summary: str) -> str:
    """Build a CLI field description for ``file_glob`` including CTS wildcard tokens.

    :param summary: short description of matched input files
    :type summary: str
    :return: field description with CTS wildcard token reference
    :rtype: str
    """
    base = summary.rstrip(".")
    return f"{base}. {CTS_GLOB_WILDCARDS_DESCRIPTION}"