PyArrow to JSON Schema Specification¶

This document specifies how a PyArrow pa.Schema determines:

1. The Draft 2020-12 JSON Schema document produced by generate_json_schema (and PyArrowToJSONSchema), including physical integer bounds, fixed-point decimal patterns, Base64 binary encodings, key/value map arrays, and lossless x-pyarrow extension metadata.
2. The serialized JSON row dictionaries produced by serialize_table_for_jsonschema and serialize_record_for_jsonschema from Arrow pa.Tables or Python row mappings, ready for Draft 2020-12 JSON Schema validation.
3. The errors raised for Arrow schemas or data types that have no faithful Draft 2020-12 JSON Schema representation.

Contents: 1. Conversion Rules · 2. Cases · 3. Errors · 4. Fixture Index

1. Conversion Rules¶

1.1 Top-Level Schema Resolution and Envelope¶

• R-SCHEMA-INPUT: The input to generate_json_schema must be a concrete pyarrow.Schema instance; passing a bare pa.DataType, pa.Field, pa.Table, or Python mapping raises E1.

Cases: 01_integers_floats_and_booleans, checkm2_quality_report

• R-NON-EMPTY-ROOT: The root pa.Schema must contain at least one field; an empty pa.schema([]) raises E2.

Cases: 01_integers_floats_and_booleans

• R-DIALECT: Standalone documents always emit "$schema": "https://json-schema.org/draft/2020-12/schema". When destination is supplied to generate_json_schema, the converted schema (requiring a non-empty schema_name or metadata[b"title"]) is inserted under destination["$defs"][<name>] with the inner "$schema" keyword stripped.

Cases: 01_integers_floats_and_booleans, 05_metadata_omission_and_overrides, checkm2_quality_report

• R-ENVELOPE-ID-TITLE: Top-level "title" resolves from explicit schema_name when provided, otherwise from non-empty UTF-8 schema.metadata[b"title"], and is omitted if neither is set. Top-level "$id" resolves from explicit schema_id when provided, otherwise from non-empty UTF-8 schema.metadata[b"$id"], and is omitted if neither is set.

Cases: 04_nullability_and_metadata, 05_metadata_omission_and_overrides, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-ENVELOPE-CLOSED: Every root schema and nested pa.struct emits "type": "object", "properties": {...}, and "additionalProperties": false. Top-level keywords appear in canonical order: $schema, $id, title, description, type, properties, additionalProperties, required, x-pyarrow.

Cases: 01_integers_floats_and_booleans, 04_nullability_and_metadata, 05_metadata_omission_and_overrides, checkm2_quality_report

1.2 Field Naming, Ordering, and Uniqueness¶

• R-FIELD-NAME: Every pa.Field.name in a root pa.Schema or nested pa.StructType is preserved verbatim as its property key in "properties" and (when non-nullable) in "required".

Cases: 01_integers_floats_and_booleans, 04_nullability_and_metadata, checkm2_quality_report

• R-FIELD-ORDER: "properties" keys and "required" entries preserve the exact declaration order of the source pa.Schema or pa.StructType. When every field of an object has nullable=True, "required" is omitted rather than emitted as [].

Cases: 01_integers_floats_and_booleans, 04_nullability_and_metadata, checkm2_quality_report

• R-UNIQUE-FIELD-NAMES: Sibling fields within a pa.Schema or pa.StructType must have distinct names; duplicate sibling field names raise E3.

Cases: 01_integers_floats_and_booleans

1.3 Structural Container Shapes¶

• R-SHAPE-STRUCT: A nested pa.struct([...]) with at least one child field maps to a closed JSON Schema "object" ("additionalProperties": false) carrying "x-pyarrow": {"arrow_type": "struct"}. An empty nested pa.struct([]) raises E2.

Cases: 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-SHAPE-LIST: Variable-length Arrow list types (pa.list_, pa.large_list, pa.list_view, pa.large_list_view) map to {"type": "array", "items": <item schema>} with "x-pyarrow": {"arrow_type": "<variant>"} ("list", "large_list", "list_view", or "large_list_view"). Multi-dimensional lists (list<list<T>>) nest "array" schemas recursively.

Cases: 06_structs_and_lists, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-SHAPE-FIXED-LIST: Fixed-length pa.list_(item, list_size) (pa.FixedSizeListType) maps to {"type": "array", "items": <item schema>, "minItems": list_size, "maxItems": list_size} with "x-pyarrow": {"arrow_type": "fixed_size_list", "list_size": list_size}.

Cases: 06_structs_and_lists, uniref_cluster_entry

• R-SHAPE-MAP: pa.map_(key_type, item_type, keys_sorted=...) maps to an "array" of closed key/value entry objects ("properties": {"key": <key schema>, "value": <value schema>}, "required": ["key", "value"], "additionalProperties": false), carrying "description": "PyArrow map rendered as key/value pairs (JSON object keys must be strings)." and "x-pyarrow": {"arrow_type": "map", "keys_sorted": <bool>}, supporting both string and non-string Arrow key types.

Cases: 07_maps_and_dictionaries, checkm2_quality_report, uniref_cluster_entry

• R-SHAPE-DICTIONARY: pa.dictionary(index_type, value_type, ordered=...) emits the JSON Schema core type and constraints of value_type while recording "arrow_type": "dictionary", "value_type", "index_type", and "ordered" inside "x-pyarrow".

Cases: 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary

1.4 Field and Element Nullability¶

• R-NULL-REQUIRED: A field with pa.Field.nullable == False is included in its enclosing object's "required" array and its property subschema is emitted directly without a null union wrapper.

Cases: 01_integers_floats_and_booleans, 02_strings_binary_and_decimals, 03_temporal_types, 04_nullability_and_metadata, 05_metadata_omission_and_overrides, 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-NULL-OPTIONAL: A field with pa.Field.nullable == True is omitted from "required" and its value subschema (including its type-level "x-pyarrow" block) is wrapped in {"anyOf": [<non-null schema>, {"type": "null"}]}. Standalone pa.null() columns or element types have no value domain and raise E4.

Cases: 02_strings_binary_and_decimals, 03_temporal_types, 04_nullability_and_metadata, 05_metadata_omission_and_overrides, 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-NULL-LIST-ITEMS: A list type's inner value_field.nullable controls element nullability independently of the outer list field: value_field.nullable == False emits <item schema> directly under "items", whereas value_field.nullable == True wraps "items" in {"anyOf": [<item schema>, {"type": "null"}]}.

Cases: 06_structs_and_lists

• R-NULL-MAP-VALUES: Arrow map keys are always non-nullable ("key" has no null union). When MapType.item_field.nullable is True, the entry object's "value" property is wrapped in {"anyOf": [<value schema>, {"type": "null"}]} while both "key" and "value" remain in the entry object's "required" array.

Cases: 07_maps_and_dictionaries

1.5 Scalar Type Mappings¶

• R-SCALAR-BOOL: pa.bool_() maps to {"type": "boolean"} with "x-pyarrow": {"arrow_type": "bool"}.

Cases: 01_integers_floats_and_booleans, ncbi_assembly_summary

• R-SCALAR-SIGNED-INT: Signed integer types (int8, int16, int32, int64) map to "type": "integer" with exact two's-complement bounds (minimum: -2^(w-1), maximum: 2^(w-1) - 1 for bit width w) and "x-pyarrow": {"arrow_type": "int8"|"int16"|"int32"|"int64"}.

Cases: 01_integers_floats_and_booleans, checkm2_quality_report, ncbi_assembly_summary

• R-SCALAR-UNSIGNED-INT: Unsigned integer types (uint8, uint16, uint32, uint64) map to "type": "integer" with minimum: 0, maximum: 2^w - 1 for bit width w, and "x-pyarrow": {"arrow_type": "uint8"|"uint16"|"uint32"|"uint64"}.

Cases: 01_integers_floats_and_booleans, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-SCALAR-FLOAT: IEEE-754 floating-point types (float16, float32, float64) map to {"type": "number"} with "x-pyarrow": {"arrow_type": "float16"|"float32"|"float64"}.

Cases: 01_integers_floats_and_booleans, checkm2_quality_report, uniref_cluster_entry

• R-SCALAR-DECIMAL: pa.decimal128(p, s) and pa.decimal256(p, s) with 0 <= s <= p map to {"type": "string", "pattern": "<fixed-point regex>"} enforcing at most p - s integer digits and s fractional digits, plus "x-pyarrow": {"arrow_type": "decimal128"|"decimal256", "logical_type": "decimal", "precision": p, "scale": s}. Decimals with s < 0 or s > p raise E5.

Cases: 02_strings_binary_and_decimals, checkm2_quality_report, ncbi_assembly_summary

• R-SCALAR-STRING: UTF-8 string types (string, large_string, string_view) map to {"type": "string"} with "x-pyarrow": {"arrow_type": "string"|"large_string"|"string_view"}.

Cases: 02_strings_binary_and_decimals, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-SCALAR-BINARY: Variable-length binary types (binary, large_binary, binary_view) map to {"type": "string", "contentEncoding": "base64"} with "x-pyarrow": {"arrow_type": "binary"|"large_binary"|"binary_view"}. Fixed-width pa.binary(n) (FixedSizeBinaryType) maps to {"type": "string", "contentEncoding": "base64"} with "x-pyarrow": {"arrow_type": "fixed_size_binary", "byte_width": n}.

Cases: 02_strings_binary_and_decimals, ncbi_assembly_summary, uniref_cluster_entry

• R-SCALAR-DATE-TIME: date32 and date64 map to {"type": "string", "format": "date"} (arrow_type: "date32" or "date64"). time32("s"|"ms") and time64("us"|"ns") map to {"type": "string", "format": "time"} with "x-pyarrow": {"arrow_type": "time32"|"time64", "unit": "<unit>"}. Relative durations and calendar intervals (pa.duration, pa.month_day_nano_interval) raise E6.

Cases: 03_temporal_types, ncbi_assembly_summary

• R-SCALAR-TIMESTAMP: Timezone-aware pa.timestamp(unit, tz=...) (non-empty tz) maps to {"type": "string", "format": "date-time"} with "x-pyarrow": {"arrow_type": "timestamp", "logical_type": "timestamp-with-tz", "unit": "<unit>", "tz": "<tz>"}. Timezone-naive pa.timestamp(unit) (tz=None) omits "format", sets "description": "ISO 8601 timestamp with no timezone attached." on the scalar node, and records "x-pyarrow": {"arrow_type": "timestamp", "logical_type": "timestamp-without-tz", "unit": "<unit>"}. Arrow unions (sparse_union, dense_union) raise E7; run_end_encoded and BaseExtensionType types (pa.uuid(), custom ExtensionType) raise E8.

Cases: 03_temporal_types, ncbi_assembly_summary, uniref_cluster_entry

1.6 Descriptions and x-pyarrow Extensions¶

• R-META-DESCRIPTION: Non-empty UTF-8 metadata[b"comment"] on a pa.Schema, pa.Field, list value_field, or map key_field/item_field is emitted as JSON Schema "description" and excluded from x-pyarrow.metadata. On nullable fields (nullable=True), field-level "description" and field-level "x-pyarrow" metadata are placed on the outer property schema outside "anyOf".

Cases: 04_nullability_and_metadata, 05_metadata_omission_and_overrides, 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-META-X-PYARROW-TYPE: When include_pyarrow_metadata=True (the default), every converted scalar, dictionary, struct, list, and map schema node carries a validated "x-pyarrow" object recording arrow_type and applicable parametric fields (logical_type, precision, scale, unit, tz, byte_width, list_size, keys_sorted, index_type, value_type, ordered).

Cases: 01_integers_floats_and_booleans, 02_strings_binary_and_decimals, 03_temporal_types, 04_nullability_and_metadata, 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-META-X-PYARROW-CUSTOM: Custom UTF-8 key/value entries in pa.Schema.metadata (excluding comment, title, $id), pa.Field.metadata (excluding comment), and list/map child field metadata are preserved under "x-pyarrow": {"metadata": {...}} when include_pyarrow_metadata=True.

Cases: 04_nullability_and_metadata, 06_structs_and_lists, uniref_cluster_entry

• R-META-OMIT-FLAG: Setting include_pyarrow_metadata=False omits all "x-pyarrow" blocks at the document, field, and nested node levels, yielding a pure Draft 2020-12 JSON Schema while retaining "$id", "title", and "description".

Cases: 05_metadata_omission_and_overrides

1.7 Arrow Table and Record Serialization¶

• R-SER-ROW-ORDER: serialize_table_for_jsonschema and serialize_record_for_jsonschema emit one JSON object per row with keys in pa.Schema declaration order; null Arrow scalars serialize as null (None), structs serialize as nested dicts, lists serialize as JSON arrays, and dictionary scalars unwrap to their decoded value.

Cases: 01_integers_floats_and_booleans, 02_strings_binary_and_decimals, 03_temporal_types, 04_nullability_and_metadata, 05_metadata_omission_and_overrides, 06_structs_and_lists, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary, uniref_cluster_entry

• R-SER-DECIMAL-STR: decimal128 and decimal256 scalars serialize via plain fixed-point formatting (format(value, "f"), never scientific notation) matching the emitted pattern regex.

Cases: 02_strings_binary_and_decimals, 07_maps_and_dictionaries, checkm2_quality_report, ncbi_assembly_summary

• R-SER-BASE64: binary, large_binary, binary_view, and fixed_size_binary(n) scalars serialize to ASCII Base64 strings matching "contentEncoding": "base64".

Cases: 02_strings_binary_and_decimals, ncbi_assembly_summary, uniref_cluster_entry

• R-SER-TEMPORAL-ISO: date32 and date64 serialize as YYYY-MM-DD; time32 and time64 serialize as HH:MM:SS[.fffffffff] (preserving sub-microsecond nanosecond digits when non-zero); timezone-aware timestamp serializes with its explicit ISO-8601 UTC/IANA offset; timezone-naive timestamp serializes without a timezone suffix.

Cases: 03_temporal_types, ncbi_assembly_summary, uniref_cluster_entry

• R-SER-MAP-ENTRIES: pa.map_ scalars serialize as [{"key": <serialized key>, "value": <serialized value>}, ...] preserving entry order and typed non-string keys.

Cases: 07_maps_and_dictionaries, checkm2_quality_report, uniref_cluster_entry

• R-SER-FINITE-FLOAT: Finite float16, float32, and float64 scalars serialize as JSON float numbers; non-finite NaN or ±Infinity values raise ValueError.

Cases: 01_integers_floats_and_booleans

2. Cases¶

Booleans, Signed and Unsigned Integers, and IEEE-754 Floats (spec/01_integers_floats_and_booleans)¶

Rules: R-SCHEMA-INPUT, R-NON-EMPTY-ROOT, R-DIALECT, R-ENVELOPE-CLOSED, R-FIELD-NAME, R-FIELD-ORDER, R-UNIQUE-FIELD-NAMES, R-NULL-REQUIRED, R-SCALAR-BOOL, R-SCALAR-SIGNED-INT, R-SCALAR-UNSIGNED-INT, R-SCALAR-FLOAT, R-META-X-PYARROW-TYPE, R-SER-ROW-ORDER, R-SER-FINITE-FLOAT

Exercises exact two's-complement minimum/maximum bounds for int8..int64, unsigned 0..2^w - 1 bounds for uint8..uint64 (including full 64-bit 18446744073709551615), bool_, and float16/float32/float64 at boundary values.

Schema (schema.py)¶

"""PyArrow schema for 01_integers_floats_and_booleans."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field("flag", pa.bool_(), nullable=False),
        pa.field("i8", pa.int8(), nullable=False),
        pa.field("i16", pa.int16(), nullable=False),
        pa.field("i32", pa.int32(), nullable=False),
        pa.field("i64", pa.int64(), nullable=False),
        pa.field("u8", pa.uint8(), nullable=False),
        pa.field("u16", pa.uint16(), nullable=False),
        pa.field("u32", pa.uint32(), nullable=False),
        pa.field("u64", pa.uint64(), nullable=False),
        pa.field("f16", pa.float16(), nullable=False),
        pa.field("f32", pa.float32(), nullable=False),
        pa.field("f64", pa.float64(), nullable=False),
    ]
)

Input (table.py)¶

"""Sample PyArrow table for 01_integers_floats_and_booleans."""

import numpy as np
import pyarrow as pa
from schema import SCHEMA

TABLE = pa.Table.from_pylist(
    [
        {
            "flag": True,
            "i8": -128,
            "i16": -32768,
            "i32": -2147483648,
            "i64": -9223372036854775808,
            "u8": 0,
            "u16": 0,
            "u32": 0,
            "u64": 0,
            "f16": np.float16(-1.5),
            "f32": -0.25,
            "f64": -123456.7890125,
        },
        {
            "flag": False,
            "i8": 127,
            "i16": 32767,
            "i32": 2147483647,
            "i64": 9223372036854775807,
            "u8": 255,
            "u16": 65535,
            "u32": 4294967295,
            "u64": 18446744073709551615,
            "f16": np.float16(3.5),
            "f32": 2.5,
            "f64": 0.0,
        },
    ],
    schema=SCHEMA,
)

Serialized records: run boundary_values¶

[
  {
    "flag": true,
    "i8": -128,
    "i16": -32768,
    "i32": -2147483648,
    "i64": -9223372036854775808,
    "u8": 0,
    "u16": 0,
    "u32": 0,
    "u64": 0,
    "f16": -1.5,
    "f32": -0.25,
    "f64": -123456.7890125
  },
  {
    "flag": false,
    "i8": 127,
    "i16": 32767,
    "i32": 2147483647,
    "i64": 9223372036854775807,
    "u8": 255,
    "u16": 65535,
    "u32": 4294967295,
    "u64": 18446744073709551615,
    "f16": 3.5,
    "f32": 2.5,
    "f64": 0.0
  }
]

JSON Schema (expected_schema.json)¶

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "flag": {
      "type": "boolean",
      "x-pyarrow": {
        "arrow_type": "bool"
      }
    },
    "i8": {
      "type": "integer",
      "minimum": -128,
      "maximum": 127,
      "x-pyarrow": {
        "arrow_type": "int8"
      }
    },
    "i16": {
      "type": "integer",
      "minimum": -32768,
      "maximum": 32767,
      "x-pyarrow": {
        "arrow_type": "int16"
      }
    },
    "i32": {
      "type": "integer",
      "minimum": -2147483648,
      "maximum": 2147483647,
      "x-pyarrow": {
        "arrow_type": "int32"
      }
    },
    "i64": {
      "type": "integer",
      "minimum": -9223372036854775808,
      "maximum": 9223372036854775807,
      "x-pyarrow": {
        "arrow_type": "int64"
      }
    },
    "u8": {
      "type": "integer",
      "minimum": 0,
      "maximum": 255,
      "x-pyarrow": {
        "arrow_type": "uint8"
      }
    },
    "u16": {
      "type": "integer",
      "minimum": 0,
      "maximum": 65535,
      "x-pyarrow": {
        "arrow_type": "uint16"
      }
    },
    "u32": {
      "type": "integer",
      "minimum": 0,
      "maximum": 4294967295,
      "x-pyarrow": {
        "arrow_type": "uint32"
      }
    },
    "u64": {
      "type": "integer",
      "minimum": 0,
      "maximum": 18446744073709551615,
      "x-pyarrow": {
        "arrow_type": "uint64"
      }
    },
    "f16": {
      "type": "number",
      "x-pyarrow": {
        "arrow_type": "float16"
      }
    },
    "f32": {
      "type": "number",
      "x-pyarrow": {
        "arrow_type": "float32"
      }
    },
    "f64": {
      "type": "number",
      "x-pyarrow": {
        "arrow_type": "float64"
      }
    }
  },
  "additionalProperties": false,
  "required": [
    "flag",
    "i8",
    "i16",
    "i32",
    "i64",
    "u8",
    "u16",
    "u32",
    "u64",
    "f16",
    "f32",
    "f64"
  ]
}

Strings, Binary Encodings, and Fixed-Point Decimals (spec/02_strings_binary_and_decimals)¶

Rules: R-SCALAR-STRING, R-SCALAR-BINARY, R-SCALAR-DECIMAL, R-NULL-REQUIRED, R-NULL-OPTIONAL, R-META-X-PYARROW-TYPE, R-SER-ROW-ORDER, R-SER-DECIMAL-STR, R-SER-BASE64

Covers UTF-8 string variants (string, large_string, string_view), Base64-encoded binary variants (binary, large_binary, binary_view, fixed_size_binary), and regex-constrained fixed-point decimals (decimal128 with scale < precision, scale == precision, scale == 0, and decimal256 including tiny values formatted without scientific notation).

Schema (schema.py)¶

"""PyArrow schema for 02_strings_binary_and_decimals."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field("utf8_str", pa.string(), nullable=False),
        pa.field("large_str", pa.large_string(), nullable=False),
        pa.field("view_str", pa.string_view(), nullable=False),
        pa.field("raw_bin", pa.binary(), nullable=False),
        pa.field("large_bin", pa.large_binary(), nullable=False),
        pa.field("view_bin", pa.binary_view(), nullable=False),
        pa.field("fixed_bin_4", pa.binary(4), nullable=False),
        pa.field("dec_standard", pa.decimal128(10, 2), nullable=False),
        pa.field("dec_fraction_only", pa.decimal128(4, 4), nullable=False),
        pa.field("dec_integer_only", pa.decimal128(8, 0), nullable=False),
        pa.field("dec256_high_prec", pa.decimal256(50, 10), nullable=True),
    ]
)

Input (table.py)¶

"""Sample PyArrow table for 02_strings_binary_and_decimals."""

from decimal import Decimal

import pyarrow as pa
from schema import SCHEMA

TABLE = pa.Table.from_pylist(
    [
        {
            "utf8_str": "contig_0001",
            "large_str": "ATGCNATGCNATGCN",
            "view_str": "short-view",
            "raw_bin": b"\x00\x01\x02\xff",
            "large_bin": b"kbase-large-payload",
            "view_bin": b"binary-view-bytes",
            "fixed_bin_4": b"\xde\xad\xbe\xef",
            "dec_standard": Decimal("-12345678.95"),
            "dec_fraction_only": Decimal("0.0625"),
            "dec_integer_only": Decimal(98765432),
            "dec256_high_prec": Decimal("0.0000000001"),
        },
        {
            "utf8_str": "",
            "large_str": "second-large-string",
            "view_str": "inlined-or-out-of-line-string-view-value",
            "raw_bin": b"",
            "large_bin": b"\x89PNG",
            "view_bin": b"1234567890abcdef",
            "fixed_bin_4": b"\x00\x00\x00\x01",
            "dec_standard": Decimal("0.00"),
            "dec_fraction_only": Decimal("-0.9999"),
            "dec_integer_only": Decimal(-42),
            "dec256_high_prec": None,
        },
    ],
    schema=SCHEMA,
)

Serialized records: run default¶

[
  {
    "utf8_str": "contig_0001",
    "large_str": "ATGCNATGCNATGCN",
    "view_str": "short-view",
    "raw_bin": "AAEC/w==",
    "large_bin": "a2Jhc2UtbGFyZ2UtcGF5bG9hZA==",
    "view_bin": "YmluYXJ5LXZpZXctYnl0ZXM=",
    "fixed_bin_4": "3q2+7w==",
    "dec_standard": "-12345678.95",
    "dec_fraction_only": "0.0625",
    "dec_integer_only": "98765432",
    "dec256_high_prec": "0.0000000001"
  },
  {
    "utf8_str": "",
    "large_str": "second-large-string",
    "view_str": "inlined-or-out-of-line-string-view-value",
    "raw_bin": "",
    "large_bin": "iVBORw==",
    "view_bin": "MTIzNDU2Nzg5MGFiY2RlZg==",
    "fixed_bin_4": "AAAAAQ==",
    "dec_standard": "0.00",
    "dec_fraction_only": "-0.9999",
    "dec_integer_only": "-42",
    "dec256_high_prec": null
  }
]

JSON Schema (expected_schema.json)¶

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "utf8_str": {
      "type": "string",
      "x-pyarrow": {
        "arrow_type": "string"
      }
    },
    "large_str": {
      "type": "string",
      "x-pyarrow": {
        "arrow_type": "large_string"
      }
    },
    "view_str": {
      "type": "string",
      "x-pyarrow": {
        "arrow_type": "string_view"
      }
    },
    "raw_bin": {
      "type": "string",
      "contentEncoding": "base64",
      "x-pyarrow": {
        "arrow_type": "binary"
      }
    },
    "large_bin": {
      "type": "string",
      "contentEncoding": "base64",
      "x-pyarrow": {
        "arrow_type": "large_binary"
      }
    },
    "view_bin": {
      "type": "string",
      "contentEncoding": "base64",
      "x-pyarrow": {
        "arrow_type": "binary_view"
      }
    },
    "fixed_bin_4": {
      "type": "string",
      "contentEncoding": "base64",
      "x-pyarrow": {
        "arrow_type": "fixed_size_binary",
        "byte_width": 4
      }
    },
    "dec_standard": {
      "type": "string",
      "pattern": "^-?\\d{1,8}(\\.\\d{1,2})?$",
      "x-pyarrow": {
        "arrow_type": "decimal128",
        "logical_type": "decimal",
        "precision": 10,
        "scale": 2
      }
    },
    "dec_fraction_only": {
      "type": "string",
      "pattern": "^-?0(\\.\\d{1,4})?$",
      "x-pyarrow": {
        "arrow_type": "decimal128",
        "logical_type": "decimal",
        "precision": 4,
        "scale": 4
      }
    },
    "dec_integer_only": {
      "type": "string",
      "pattern": "^-?\\d{1,8}$",
      "x-pyarrow": {
        "arrow_type": "decimal128",
        "logical_type": "decimal",
        "precision": 8,
        "scale": 0
      }
    },
    "dec256_high_prec": {
      "anyOf": [
        {
          "type": "string",
          "pattern": "^-?\\d{1,40}(\\.\\d{1,10})?$",
          "x-pyarrow": {
            "arrow_type": "decimal256",
            "logical_type": "decimal",
            "precision": 50,
            "scale": 10
          }
        },
        {
          "type": "null"
        }
      ]
    }
  },
  "additionalProperties": false,
  "required": [
    "utf8_str",
    "large_str",
    "view_str",
    "raw_bin",
    "large_bin",
    "view_bin",
    "fixed_bin_4",
    "dec_standard",
    "dec_fraction_only",
    "dec_integer_only"
  ]
}

Dates, Times, and Timezone-Aware vs Timezone-Naive Timestamps (spec/03_temporal_types)¶

Rules: R-SCALAR-DATE-TIME, R-SCALAR-TIMESTAMP, R-NULL-REQUIRED, R-NULL-OPTIONAL, R-META-X-PYARROW-TYPE, R-SER-ROW-ORDER, R-SER-TEMPORAL-ISO

Covers date32, date64, time32("s"|"ms"), time64("us"|"ns") (including sub-microsecond nanoseconds), timezone-aware timestamp across units (s, ms, us, ns) and IANA/UTC zones ("format": "date-time"), and timezone-naive timestamp (logical_type: "timestamp-without-tz" without "format": "date-time").

Schema (schema.py)¶

"""PyArrow schema for 03_temporal_types."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field("d32", pa.date32(), nullable=False),
        pa.field("d64", pa.date64(), nullable=False),
        pa.field("t32_s", pa.time32("s"), nullable=False),
        pa.field("t32_ms", pa.time32("ms"), nullable=False),
        pa.field("t64_us", pa.time64("us"), nullable=False),
        pa.field("t64_ns", pa.time64("ns"), nullable=False),
        pa.field("ts_s_utc", pa.timestamp("s", tz="UTC"), nullable=False),
        pa.field("ts_ms_utc", pa.timestamp("ms", tz="UTC"), nullable=False),
        pa.field("ts_us_la", pa.timestamp("us", tz="America/Los_Angeles"), nullable=False),
        pa.field("ts_ns_utc", pa.timestamp("ns", tz="UTC"), nullable=False),
        pa.field("ts_naive_us", pa.timestamp("us"), nullable=False),
        pa.field("ts_naive_ns_opt", pa.timestamp("ns"), nullable=True),
    ]
)

Input (table.py)¶

"""Sample PyArrow table for 03_temporal_types."""

import datetime as dt

import pyarrow as pa
from schema import SCHEMA

TABLE = pa.table(
    {
        "d32": pa.array([dt.date(2026, 4, 10), dt.date(1970, 1, 1)], type=pa.date32()),
        "d64": pa.array([dt.date(2026, 4, 11), dt.date(2000, 2, 29)], type=pa.date64()),
        "t32_s": pa.array([dt.time(8, 30, 0), dt.time(23, 59, 59)], type=pa.time32("s")),
        "t32_ms": pa.array([dt.time(8, 30, 0, 250000), dt.time(0, 0, 0)], type=pa.time32("ms")),
        "t64_us": pa.array([dt.time(14, 15, 30, 123456), dt.time(12, 0, 0)], type=pa.time64("us")),
        "t64_ns": pa.array([52_200_123_456_789, 0], type=pa.time64("ns")),
        "ts_s_utc": pa.array(
            [
                dt.datetime(2026, 4, 10, 14, 30, 0, tzinfo=dt.UTC),
                dt.datetime(2025, 1, 1, 0, 0, 0, tzinfo=dt.UTC),
            ],
            type=pa.timestamp("s", tz="UTC"),
        ),
        "ts_ms_utc": pa.array(
            [
                dt.datetime(2026, 4, 10, 14, 30, 0, 500000, tzinfo=dt.UTC),
                dt.datetime(2025, 1, 1, 0, 0, 0, tzinfo=dt.UTC),
            ],
            type=pa.timestamp("ms", tz="UTC"),
        ),
        "ts_us_la": pa.array(
            [
                dt.datetime(2026, 4, 10, 14, 30, 0, 125000, tzinfo=dt.UTC),
                dt.datetime(2025, 1, 15, 18, 0, 0, tzinfo=dt.UTC),
            ],
            type=pa.timestamp("us", tz="America/Los_Angeles"),
        ),
        "ts_ns_utc": pa.array(
            [
                dt.datetime(2026, 4, 10, 14, 30, 0, 750000, tzinfo=dt.UTC),
                dt.datetime(2025, 6, 1, 6, 0, 0, tzinfo=dt.UTC),
            ],
            type=pa.timestamp("ns", tz="UTC"),
        ),
        "ts_naive_us": pa.array(
            [
                dt.datetime(2026, 4, 10, 9, 45, 0, tzinfo=dt.UTC).replace(tzinfo=None),
                dt.datetime(2025, 12, 31, 23, 59, 59, tzinfo=dt.UTC).replace(tzinfo=None),
            ],
            type=pa.timestamp("us"),
        ),
        "ts_naive_ns_opt": pa.array(
            [
                dt.datetime(2026, 4, 10, 9, 45, 0, 500000, tzinfo=dt.UTC).replace(tzinfo=None),
                None,
            ],
            type=pa.timestamp("ns"),
        ),
    },
    schema=SCHEMA,
)

Serialized records: run default¶

[
  {
    "d32": "2026-04-10",
    "d64": "2026-04-11",
    "t32_s": "08:30:00",
    "t32_ms": "08:30:00.250000",
    "t64_us": "14:15:30.123456",
    "t64_ns": "14:30:00.123456789",
    "ts_s_utc": "2026-04-10T14:30:00+00:00",
    "ts_ms_utc": "2026-04-10T14:30:00.500000+00:00",
    "ts_us_la": "2026-04-10T07:30:00.125000-07:00",
    "ts_ns_utc": "2026-04-10T14:30:00.750000+00:00",
    "ts_naive_us": "2026-04-10T09:45:00",
    "ts_naive_ns_opt": "2026-04-10T09:45:00.500000"
  },
  {
    "d32": "1970-01-01",
    "d64": "2000-02-29",
    "t32_s": "23:59:59",
    "t32_ms": "00:00:00",
    "t64_us": "12:00:00",
    "t64_ns": "00:00:00",
    "ts_s_utc": "2025-01-01T00:00:00+00:00",
    "ts_ms_utc": "2025-01-01T00:00:00+00:00",
    "ts_us_la": "2025-01-15T10:00:00-08:00",
    "ts_ns_utc": "2025-06-01T06:00:00+00:00",
    "ts_naive_us": "2025-12-31T23:59:59",
    "ts_naive_ns_opt": null
  }
]

JSON Schema (expected_schema.json)¶

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "d32": {
      "type": "string",
      "format": "date",
      "x-pyarrow": {
        "arrow_type": "date32"
      }
    },
    "d64": {
      "type": "string",
      "format": "date",
      "x-pyarrow": {
        "arrow_type": "date64"
      }
    },
    "t32_s": {
      "type": "string",
      "format": "time",
      "x-pyarrow": {
        "arrow_type": "time32",
        "unit": "s"
      }
    },
    "t32_ms": {
      "type": "string",
      "format": "time",
      "x-pyarrow": {
        "arrow_type": "time32",
        "unit": "ms"
      }
    },
    "t64_us": {
      "type": "string",
      "format": "time",
      "x-pyarrow": {
        "arrow_type": "time64",
        "unit": "us"
      }
    },
    "t64_ns": {
      "type": "string",
      "format": "time",
      "x-pyarrow": {
        "arrow_type": "time64",
        "unit": "ns"
      }
    },
    "ts_s_utc": {
      "type": "string",
      "format": "date-time",
      "x-pyarrow": {
        "arrow_type": "timestamp",
        "logical_type": "timestamp-with-tz",
        "unit": "s",
        "tz": "UTC"
      }
    },
    "ts_ms_utc": {
      "type": "string",
      "format": "date-time",
      "x-pyarrow": {
        "arrow_type": "timestamp",
        "logical_type": "timestamp-with-tz",
        "unit": "ms",
        "tz": "UTC"
      }
    },
    "ts_us_la": {
      "type": "string",
      "format": "date-time",
      "x-pyarrow": {
        "arrow_type": "timestamp",
        "logical_type": "timestamp-with-tz",
        "unit": "us",
        "tz": "America/Los_Angeles"
      }
    },
    "ts_ns_utc": {
      "type": "string",
      "format": "date-time",
      "x-pyarrow": {
        "arrow_type": "timestamp",
        "logical_type": "timestamp-with-tz",
        "unit": "ns",
        "tz": "UTC"
      }
    },
    "ts_naive_us": {
      "type": "string",
      "description": "ISO 8601 timestamp with no timezone attached.",
      "x-pyarrow": {
        "arrow_type": "timestamp",
        "logical_type": "timestamp-without-tz",
        "unit": "us"
      }
    },
    "ts_naive_ns_opt": {
      "anyOf": [
        {
          "type": "string",
          "description": "ISO 8601 timestamp with no timezone attached.",
          "x-pyarrow": {
            "arrow_type": "timestamp",
            "logical_type": "timestamp-without-tz",
            "unit": "ns"
          }
        },
        {
          "type": "null"
        }
      ]
    }
  },
  "additionalProperties": false,
  "required": [
    "d32",
    "d64",
    "t32_s",
    "t32_ms",
    "t64_us",
    "t64_ns",
    "ts_s_utc",
    "ts_ms_utc",
    "ts_us_la",
    "ts_ns_utc",
    "ts_naive_us"
  ]
}

Nullability, Envelope Metadata, Descriptions, and Custom Arrow Metadata (spec/04_nullability_and_metadata)¶

Rules: R-ENVELOPE-ID-TITLE, R-ENVELOPE-CLOSED, R-FIELD-NAME, R-FIELD-ORDER, R-NULL-REQUIRED, R-NULL-OPTIONAL, R-META-DESCRIPTION, R-META-X-PYARROW-TYPE, R-META-X-PYARROW-CUSTOM, R-SER-ROW-ORDER

Demonstrates resolution of "title", "$id", and "description" from pa.Schema.metadata, preservation of custom schema and field metadata under x-pyarrow.metadata, placement of field "description" and x-pyarrow.metadata outside "anyOf" on nullable fields, and omission of "required" on a nested struct whose fields are all nullable.

Schema (schema.py)¶

"""PyArrow schema for 04_nullability_and_metadata."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field(
            "sample_id",
            pa.string(),
            nullable=False,
            metadata={
                "comment": "Unique biosample identifier.",
                "pydantic_type": "string",
                "source_column": "biosample_acc",
            },
        ),
        pa.field(
            "depth_m",
            pa.float64(),
            nullable=True,
            metadata={
                "comment": "Sampling depth in meters below surface.",
                "unit": "m",
            },
        ),
        pa.field(
            "quality_score",
            pa.int32(),
            nullable=True,
        ),
        pa.field(
            "optional_details",
            pa.struct(
                [
                    pa.field("collector", pa.string(), nullable=True, metadata={"comment": "Field collector name."}),
                    pa.field("ph", pa.float32(), nullable=True),
                ]
            ),
            nullable=True,
            metadata={"comment": "Optional environmental annotations where every child field is nullable."},
        ),
    ],
    metadata={
        "title": "EnvironmentalSample",
        "$id": "https://kbase.us/schemas/environmental_sample.json",
        "comment": "Environmental sample record with rich Arrow metadata.",
        "dataset_version": "2026.04",
        "curator": "cdm-ingest",
    },
)

Input (table.py)¶

"""Sample PyArrow table for 04_nullability_and_metadata."""

import pyarrow as pa
from schema import SCHEMA

TABLE = pa.Table.from_pylist(
    [
        {
            "sample_id": "SAMN00010001",
            "depth_m": 125.5,
            "quality_score": 98,
            "optional_details": {"collector": "A. Smith", "ph": 7.25},
        },
        {
            "sample_id": "SAMN00010002",
            "depth_m": None,
            "quality_score": None,
            "optional_details": None,
        },
    ],
    schema=SCHEMA,
)

Serialized records: run default¶

[
  {
    "sample_id": "SAMN00010001",
    "depth_m": 125.5,
    "quality_score": 98,
    "optional_details": {
      "collector": "A. Smith",
      "ph": 7.25
    }
  },
  {
    "sample_id": "SAMN00010002",
    "depth_m": null,
    "quality_score": null,
    "optional_details": null
  }
]

JSON Schema (expected_schema.json)¶

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://kbase.us/schemas/environmental_sample.json",
  "title": "EnvironmentalSample",
  "description": "Environmental sample record with rich Arrow metadata.",
  "type": "object",
  "properties": {
    "sample_id": {
      "type": "string",
      "x-pyarrow": {
        "arrow_type": "string",
        "metadata": {
          "pydantic_type": "string",
          "source_column": "biosample_acc"
        }
      },
      "description": "Unique biosample identifier."
    },
    "depth_m": {
      "anyOf": [
        {
          "type": "number",
          "x-pyarrow": {
            "arrow_type": "float64"
          }
        },
        {
          "type": "null"
        }
      ],
      "description": "Sampling depth in meters below surface.",
      "x-pyarrow": {
        "metadata": {
          "unit": "m"
        }
      }
    },
    "quality_score": {
      "anyOf": [
        {
          "type": "integer",
          "minimum": -2147483648,
          "maximum": 2147483647,
          "x-pyarrow": {
            "arrow_type": "int32"
          }
        },
        {
          "type": "null"
        }
      ]
    },
    "optional_details": {
      "anyOf": [
        {
          "type": "object",
          "properties": {
            "collector": {
              "anyOf": [
                {
                  "type": "string",
                  "x-pyarrow": {
                    "arrow_type": "string"
                  }
                },
                {
                  "type": "null"
                }
              ],
              "description": "Field collector name."
            },
            "ph": {
              "anyOf": [
                {
                  "type": "number",
                  "x-pyarrow": {
                    "arrow_type": "float32"
                  }
                },
                {
                  "type": "null"
                }
              ]
            }
          },
          "additionalProperties": false,
          "x-pyarrow": {
            "arrow_type": "struct"
          }
        },
        {
          "type": "null"
        }
      ],
      "description": "Optional environmental annotations where every child field is nullable."
    }
  },
  "additionalProperties": false,
  "required": [
    "sample_id"
  ],
  "x-pyarrow": {
    "metadata": {
      "dataset_version": "2026.04",
      "curator": "cdm-ingest"
    }
  }
}

Explicit Envelope Overrides and Omitting x-pyarrow Metadata (spec/05_metadata_omission_and_overrides)¶

Rules: R-DIALECT, R-ENVELOPE-ID-TITLE, R-ENVELOPE-CLOSED, R-META-DESCRIPTION, R-META-OMIT-FLAG, R-NULL-REQUIRED, R-NULL-OPTIONAL, R-SER-ROW-ORDER

Verifies that explicit schema_name and schema_id parameters override metadata[b"title"] and metadata[b"$id"], and that include_pyarrow_metadata=False strips all "x-pyarrow" blocks while keeping standard Draft 2020-12 keywords ($schema, $id, title, description, minimum, maximum, pattern, format, contentEncoding).

Schema (schema.py)¶

"""PyArrow schema for 05_metadata_omission_and_overrides."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field("id", pa.string(), nullable=False, metadata={"comment": "Record identifier.", "xsd_type": "xs:ID"}),
        pa.field("count", pa.uint16(), nullable=False, metadata={"unit": "reads"}),
        pa.field("gc_ratio", pa.decimal128(5, 4), nullable=True, metadata={"comment": "GC fraction."}),
        pa.field("digest", pa.binary(8), nullable=False),
        pa.field("collected_on", pa.date32(), nullable=True),
    ],
    metadata={
        "title": "OverriddenTitle",
        "$id": "urn:overridden:sample",
        "comment": "Pure Draft 2020-12 schema with x-pyarrow suppressed.",
        "internal_tag": "drop-when-omitted",
    },
)

Input (table.py)¶

"""Sample PyArrow table for 05_metadata_omission_and_overrides."""

import datetime as dt
from decimal import Decimal

import pyarrow as pa
from schema import SCHEMA

TABLE = pa.Table.from_pylist(
    [
        {
            "id": "S-01",
            "count": 1024,
            "gc_ratio": Decimal("0.5125"),
            "digest": b"01234567",
            "collected_on": dt.date(2026, 3, 15),
        },
        {
            "id": "S-02",
            "count": 0,
            "gc_ratio": None,
            "digest": b"\x00\x01\x02\x03\x04\x05\x06\x07",
            "collected_on": None,
        },
    ],
    schema=SCHEMA,
)

Serialized records: run without_pyarrow_metadata (schema_name=CleanSample, schema_id=https://kbase.us/schemas/clean_sample.json, include_pyarrow_metadata=False)¶

[
  {
    "id": "S-01",
    "count": 1024,
    "gc_ratio": "0.5125",
    "digest": "MDEyMzQ1Njc=",
    "collected_on": "2026-03-15"
  },
  {
    "id": "S-02",
    "count": 0,
    "gc_ratio": null,
    "digest": "AAECAwQFBgc=",
    "collected_on": null
  }
]

JSON Schema (expected_schema.json)¶

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://kbase.us/schemas/clean_sample.json",
  "title": "CleanSample",
  "description": "Pure Draft 2020-12 schema with x-pyarrow suppressed.",
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "Record identifier."
    },
    "count": {
      "type": "integer",
      "minimum": 0,
      "maximum": 65535
    },
    "gc_ratio": {
      "anyOf": [
        {
          "type": "string",
          "pattern": "^-?\\d{1,1}(\\.\\d{1,4})?$"
        },
        {
          "type": "null"
        }
      ],
      "description": "GC fraction."
    },
    "digest": {
      "type": "string",
      "contentEncoding": "base64"
    },
    "collected_on": {
      "anyOf": [
        {
          "type": "string",
          "format": "date"
        },
        {
          "type": "null"
        }
      ]
    }
  },
  "additionalProperties": false,
  "required": [
    "id",
    "count",
    "digest"
  ]
}

Nested Structs, Variable/View Lists, Fixed-Size Lists, and 2D Lists (spec/06_structs_and_lists)¶

Rules: R-SHAPE-STRUCT, R-SHAPE-LIST, R-SHAPE-FIXED-LIST, R-NULL-REQUIRED, R-NULL-OPTIONAL, R-NULL-LIST-ITEMS, R-META-DESCRIPTION, R-META-X-PYARROW-TYPE, R-META-X-PYARROW-CUSTOM, R-SER-ROW-ORDER

Exercises nested pa.struct, pa.list_, pa.large_list, pa.list_view, pa.large_list_view, pa.list_(item, 3) (minItems: 3, maxItems: 3), 2D list<list<int32>>, nullable vs non-nullable list elements, and element-level comment + custom metadata on inner pa.field items.

Schema (schema.py)¶

"""PyArrow schema for 06_structs_and_lists."""

import pyarrow as pa

SCHEMA = pa.schema(
    [
        pa.field("feature_id", pa.string(), nullable=False),
        pa.field(
            "location",
            pa.struct(
                [
                    pa.field("contig", pa.string(), nullable=False),
                    pa.field("start", pa.int64(), nullable=False),
                    pa.field("end", pa.int64(), nullable=False),
                    pa.field("strand", pa.string(), nullable=True),
                ]
            ),
            nullable=False,
        ),
        pa.field(
            "go_terms",
            pa.list_(
                pa.field(
                    "item",
                    pa.string(),
                    nullable=False,
                    metadata={"comment": "Gene Ontology term accession.", "ontology": "GO"},
                )
            ),
            nullable=False,
        ),
        pa.field(
            "nullable_scores",
            pa.large_list(pa.field("item", pa.int32(), nullable=True)),
            nullable=True,
        ),
        pa.field(
            "aliases_view",
            pa.list_view(pa.field("item", pa.string(), nullable=False)),
            nullable=False,
        ),
        pa.field(
            "notes_large_view",
            pa.large_list_view(pa.field("item", pa.string(), nullable=True)),
            nullable=True,
        ),
        pa.field(
            "rgb_triplet",
            pa.list_(pa.field("item", pa.uint8(), nullable=False), 3),
            nullable=False,
        ),
        pa.field(
            "score_matrix",
            pa.list_(
                pa.field(
                    "row",
                    pa.list_(pa.field("cell", pa.int32(), nullable=False)),
                    nullable=False,
                )
            ),
            nullable=False,
        ),
    ]
)