Skip to content

cdm_data_loaders.pipelines.jsonlines.extract_pipeline

JSONL load pipeline for the KBase CTS.

Reads JSONL files. No validation occurs other than that records are valid JSON.

table_name names the destination table for all valid records; files may live anywhere under input_dir. Records that fail JSON parsing go to the _rejected table. Each rejected record includes the target table name, source file, line number, raw text, and error detail.

run_jsonlines_pipeline 

run_jsonlines_pipeline(
    settings: JsonlExtractSettings,
) -> LoadInfo | None

Run the JSONL pipeline.

Parameters:

Name Type Description Default
settings JsonlExtractSettings

pipeline configuration

 required

Returns:

Type Description
LoadInfo | None

load information for the pipeline
Source code in src/cdm_data_loaders/pipelines/jsonlines/extract_pipeline.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
def run_jsonlines_pipeline(settings: JsonlExtractSettings) -> LoadInfo | None:
    """Run the JSONL pipeline.

    :param settings: pipeline configuration
    :type settings: JsonlExtractSettings
    :return: load information for the pipeline
    :rtype: LoadInfo | None
    """
    jsonl_resource = build_filesystem_resource(settings, route_jsonl_lines, name="jsonl_reader")

    return run_pipeline(
        settings=settings,
        resource=jsonl_resource,
        **build_file_ingest_kwargs(settings, EXTRACT_PIPELINE_NAME, fallback_dataset_name=True),
    )