Skip to content

cdm_data_loaders.converters.cli.pyiceberg_to_jsonschema

CLI interface for creating a JSON Schema dump of table schemas from an Iceberg catalog.

IcebergToJsonSchemaSettings

Bases: IcebergCatalogDumpSettings

Settings for generating a set of JSONSchemas from an Iceberg data catalog.

Source code in src/cdm_data_loaders/converters/cli/pyiceberg_to_jsonschema.py 
28
29
30
31
32
33
34
35
class IcebergToJsonSchemaSettings(IcebergCatalogDumpSettings):
    """Settings for generating a set of JSONSchemas from an Iceberg data catalog."""

    model_config: SettingsConfigDict = default_settings_with_shortcuts(
        cli_prog_name=PIPELINE_NAME, cli_shortcuts=CATALOG_CLI_SHORTCUTS
    )

    group_by_namespace: bool = Field(default=False, description="Export one JSON Schema per namespace")

dump_catalog_schemas 

dump_catalog_schemas(
    settings: IcebergToJsonSchemaSettings,
) -> None

Iterate through a catalog and dump out the table schemas as JSONSchema.

A table that fails to load or convert is logged and skipped; it does not abort the dump of the remaining tables. generated_at is stamped identically on every table in a single run (it reflects the run, not each table's individual read time). An existing output file for a table is overwritten, with a warning logged first. With group_by_namespace, each nonempty namespace produces one file whose $defs holds its table schemas.

Parameters:

Name Type Description Default
settings IcebergToJsonSchemaSettings

Catalog name and output directory.

 required
Source code in src/cdm_data_loaders/converters/cli/pyiceberg_to_jsonschema.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def dump_catalog_schemas(settings: IcebergToJsonSchemaSettings) -> None:
    """Iterate through a catalog and dump out the table schemas as JSONSchema.

    A table that fails to load or convert is logged and skipped; it does not
    abort the dump of the remaining tables. ``generated_at`` is stamped
    identically on every table in a single run (it reflects the run, not each
    table's individual read time). An existing output file for a table is
    overwritten, with a warning logged first. With ``group_by_namespace``,
    each nonempty namespace produces one file whose ``$defs`` holds its table schemas.

    :param settings: Catalog name and output directory.
    :type settings: IcebergToJsonSchemaSettings
    """
    dump_catalog(settings, JSON_SCHEMA_DUMP_DRIVER, logger)