Schema Converter Specifications Developer Guide¶

This guide explains how to author, edit, and maintain specification rules, test cases, error catalogs, and generated reference pages under specs/schema_converters/.

1. Architecture and Directory Layout¶

Each schema converter (jsonschema_to_pyarrow, pydantic_to_pyarrow, xsd_to_pyarrow, pyarrow_to_jsonschema, pyarrow_to_linkml) has a specification directory under specs/schema_converters/<converter>/:

specs/schema_converters/<converter>/
├── rules.yaml                       # Rule catalogue (sections + rule IDs and statements)
├── spec/
│   ├── 01_<topic>/
│   │   ├── case.yaml                # Case manifest (title, schema, rules, runs)
│   │   ├── schema.<ext>             # Input schema (.json, .py, or .xsd)
│   │   ├── expected_schema.<ext>    # Expected schema (.py, .json, or .yaml)
│   │   ├── data.<ext> / table.py    # Input records (.json, .xml, or PyArrow Table .py)
│   │   ├── expected.json            # Expected coerced/serialized rows as JSON
│   │   └── expected_table.py        # Expected PyArrow Table (EXPECTED_TABLE / TABLE)
│   └── errors/
│       ├── manifest.yaml            # Error fixture manifest (file, code, match, description)
│       └── <error_fixture>.<ext>    # Invalid/unsupported schemas triggering E1..En
└── examples/
    └── <real_world_example>/
        └── case.yaml                # Real-world domain schema cases (PubMed, UniProt, NCBI, etc.)

The manifests in specs/schema_converters/ drive two outputs:

1. Parametrized specification tests:
2. tests/cdm_data_loaders/converters/facades/test_jsonschema_to_pyarrow_spec.py
3. tests/cdm_data_loaders/converters/facades/test_pydantic_to_pyarrow_spec.py
4. tests/cdm_data_loaders/converters/facades/test_xsd_to_pyarrow_spec.py
5. tests/cdm_data_loaders/converters/facades/test_pyarrow_to_jsonschema_spec.py
6. tests/cdm_data_loaders/converters/facades/test_pyarrow_to_linkml_spec.py
7. Generated Markdown specification pages:
8. JSON Schema to PyArrow Specification
9. Pydantic to PyArrow Specification
10. XSD to PyArrow Specification
11. PyArrow to JSON Schema Specification
12. PyArrow to LinkML Specification

2. How Rules Are Created and Structured¶

Rules live in specs/schema_converters/<converter>/rules.yaml and are loaded by load_rules() in tests/converter_spec.py.

title: XSD to PyArrow Schema and Record Conversion Specification
intro: |
  Target specification for `cdm_data_loaders.converters.facades.xsd_to_pyarrow`.

sections:
  - id: scalars
    title: Scalar Type Mapping (`XSD_SCALAR_SPEC` / `SCALAR_ARROW_TYPES`)
    intro: |
      Built-in XSD primitive and derived atomic types map to PyArrow types as follows:
    generated: xsd_scalar_table
    rules:
      - id: R-SCALAR-BUILTIN
        statement: |
          Every built-in atomic `xs:*` type maps to the PyArrow `DataType` in the table above.

• Sections: Group related rules under a title and id. A section can include generated: <key> to embed a table auto-generated from converter code (such as xsd_scalar_table, jsonschema_scalar_table, or pydantic_scalar_table in scripts/gen_converter_spec.py).
• Rule IDs: Use stable, descriptive uppercase identifiers prefixed with R- (for example, R-SCALAR-BUILTIN, R-NULL-ATTRIBUTE, R-REC-DEPTH).
• Rule statements: State one testable conversion contract per rule.
• Bidirectional enforcement:
• test_manifests_pass_every_rule_is_exercised verifies that every rule ID in rules.yaml appears in the rules: list of at least one case.yaml.
• test_manifests_pass_every_case_rule_exists verifies that every rule ID listed in any case.yaml is defined in rules.yaml.
• load_rules() rejects duplicate rule IDs.

3. How Errors Are Created and Structured¶

Converter error codes (E1, E2, ...) are defined as exception subclasses in the converter source code and paired with fixtures in spec/errors/manifest.yaml.

Source Code Error Hierarchy¶

Each converter defines a base error class and a tuple of numbered subclasses:

Each error subclass defines:

• code: unique identifier ("E1", "E2", ...).
• title: short label rendered in the specification table.
• Class docstring (__doc__): explanation of why the converter rejects the construct.

Error Manifest (spec/errors/manifest.yaml)¶

Each entry in spec/errors/manifest.yaml maps a fixture file in spec/errors/ to an error code and message regex:

errors:
  - file: wildcard_any_element.xsd
    xml_tag: entry
    code: E4
    match: xs:any
    description: "`<xs:sequence><xs:any processContents=\"lax\"/></xs:sequence>`"

Error Invariants Enforced by Tests¶

• Every class in *_ERRORS has a unique code, a non-empty title, and a docstring.
• Every code in *_ERRORS has at least one entry in spec/errors/manifest.yaml.
• Every entry in spec/errors/manifest.yaml references a valid code in *_ERRORS.
• Every schema file on disk in spec/errors/ is listed in spec/errors/manifest.yaml (test_manifests_pass_every_error_fixture_listed).
• Running the converter on each fixture raises the exact subclass ERRORS_BY_CODE[err.code] with a message matching err.match.

4. Common Developer Workflows¶

Adding a New Rule and Test Case¶

1. Add the rule to rules.yaml:
2. Add an entry with id and statement under the appropriate section in specs/schema_converters/<converter>/rules.yaml.
3. Create or update a case in spec/<case>/ (or examples/<case>/):
4. Reference the new rule ID under rules: in case.yaml.
5. Provide the input schema (schema.json, schema.py, or schema.xsd), expected_schema.py (EXPECTED_SCHEMA), input data (data.json or *.xml), expected.json, and expected_table.py (EXPECTED_TABLE).
6. Regenerate the specification page:

   uv run python -m scripts.gen_converter_spec
   

7. Run the converter spec and documentation tests:

   uv run pytest tests/cdm_data_loaders/converters/facades/
   uv run pytest -m docs
   

Adding a New Error Code¶

1. Define the exception subclass:
2. Subclass the converter's base error (XsdToPyArrowError, JSONSchemaToPyArrowError, or PydanticToPyArrowError) with code = "E<N>", title = "...", and a docstring.
3. Append the new class to XSD_TO_PYARROW_ERRORS, JSONSCHEMA_TO_PYARROW_ERRORS, or PYDANTIC_TO_PYARROW_ERRORS.
4. Raise the new subclass in the reader or facade when the invalid construct is detected.
5. Add an error fixture:
6. Place a minimal schema file reproducing the error in specs/schema_converters/<converter>/spec/errors/.
7. Add an entry in specs/schema_converters/<converter>/spec/errors/manifest.yaml with file, code, match, and description.
8. Regenerate the specification page and run tests:

   uv run python -m scripts.gen_converter_spec
   uv run pytest tests/cdm_data_loaders/converters/facades/
   uv run pytest -m docs
   

Editing or Removing an Existing Rule or Error¶

• Editing a rule statement or error docstring:
• Update the text in rules.yaml or the error subclass docstring/title.
• Run uv run python -m scripts.gen_converter_spec to sync docs/*_specification.md.
• Editing expected schema or table outputs:
• Update expected_schema.py, expected_table.py, and expected.json together so schema validation and table roundtrip assertions agree.
• Run uv run python -m scripts.gen_converter_spec to update the embedded code blocks in docs/*_specification.md.
• Removing a rule or error:
• Remove the rule ID from all case.yaml files before deleting it from rules.yaml.
• Remove the fixture file from spec/errors/ and its entry from spec/errors/manifest.yaml when removing an error subclass from *_ERRORS.
• Regenerate docs with uv run python -m scripts.gen_converter_spec.

Adding a New Converter Specification Suite¶

1. Create specs/schema_converters/<new_converter>/rules.yaml, spec/, and spec/errors/manifest.yaml.
2. Register a ConverterSpecConfig entry in CONFIGS inside scripts/gen_converter_spec.py.
3. Run uv run python -m scripts.gen_converter_spec to generate docs/<new_converter>_specification.md.
4. Register docs/<new_converter>_specification.md in zensical.toml ([project].nav), docs/index.md, and README.md.
5. Verify with uv run pytest -m docs and uv run --group docs zensical build --strict.