# Developing wireform-parquet
This guide is for maintainers working on compatibility, validation and performance.
For installation and everyday use, start with the [README](README.md). The
repository's [development guidelines](https://github.com/iand675/wireform-/blob/main/AGENTS.md) cover the toolchain,
generated sources, C harness and benchmark discipline.
## Build and test
Run these commands from the repository root:
```sh
cabal build wireform-parquet:tests \
wireform-parquet:exe:wireform-parquet-interop-probe \
wireform-parquet:exe:wireform-parquet-reverse-probe -fbrotli
cabal test wireform-parquet:tests --test-show-details=direct
```
Validate the default and minimal codec builds too; enabling every optional
codec does not check their disabled branches. Native library setup is covered
in the repository guidelines and the README.
## External compatibility checks
The scripts in [`scripts/`](scripts/) exercise the public APIs against
independent implementations, primarily Arrow C++, arrow-rs, DuckDB and Polars.
They require their respective Python/Rust dependencies and built native probes.
Obtain probe paths with `cabal list-bin` for the executable targets above.
| Area | Driver |
| --- | --- |
| Standard corpus, including malformed files | [`parquet_corpus.py`](scripts/parquet_corpus.py) |
| Compression and malformed streams | [`parquet_codec_interop.py`](scripts/parquet_codec_interop.py), [`parquet_lzo_interop.py`](scripts/parquet_lzo_interop.py) |
| Schema boundaries and projections | [`parquet_schema_interop.py`](scripts/parquet_schema_interop.py), [`parquet_projection_interop.py`](scripts/parquet_projection_interop.py) |
| Large offsets and the large-map fixture | [`parquet_large_offset_interop.py`](scripts/parquet_large_offset_interop.py), [`parquet_large_map_interop.py`](scripts/parquet_large_map_interop.py) |
| Nested values and timestamps | [`parquet_nested_types_interop.py`](scripts/parquet_nested_types_interop.py), [`parquet_timestamp_interop.py`](scripts/parquet_timestamp_interop.py) |
| Variant values and shredding | [`parquet_variant_interop.py`](scripts/parquet_variant_interop.py), [`parquet_variant_typed_interop.py`](scripts/parquet_variant_typed_interop.py), [`parquet_variant_write_interop.py`](scripts/parquet_variant_write_interop.py) |
| Encrypted writes | [`parquet_encryption_interop.py`](scripts/parquet_encryption_interop.py) |
The ordinary corpus driver uses heap, timeout and uncompressed-size limits.
Use `--include-bad-data` for adversarial fixtures and `--include-large-map` to
verify the large-map fixture through 64-bit Arrow offsets. That check needs
several GiB of RAM. It compares complete buffers using hashes and one-row C++
batches rather than constructing gigabyte Python strings.
The ordinary corpus glob does **not** include `.parquet.encrypted` fixtures.
Those require separate encryption coverage. A green ordinary-corpus run, a
successful round-trip, or a reference implementation rejecting a value does
not establish complete Parquet conformance. Record resource limits and
reference limitations explicitly, and compare actual values as well as schemas.
## Implementation and compatibility notes
Both flat writers honor compression, V1/V2 pages, requested bloom filters and
the page-index toggle. Page indexes are generated by default from the actual
page offsets, sizes and statistics. Unsupported codecs return errors instead
of silently changing the requested compression. The lower-level mixed writer
`buildParquetFileMixedWithAux` returns `Either String ByteString`; the pure
high-level writer reports invalid inputs with a descriptive exception.
`LZO` reads and writes raw LZO1X page blocks without an external LZO library
or extra framing. Both page versions and dictionary pages are supported;
V2 definition and repetition levels remain uncompressed. The shared
`Columnar.LZO` decoder checks the stream and output budget before allocating
its result. `scripts/parquet_lzo_interop.py` checks both directions against
liblzo2 and PyArrow, including nested, nullable and empty columns.
Page readers check the exact decoded size against the page header. GZip,
Snappy, Zstandard, LZ4_RAW, LZO and optional Brotli reject an output that
exceeds that destination before allocating the full expanded value stream.
All-null V2 pages may omit their value section; a nonempty section is still
validated. Empty LZ4_RAW sections use the standard final zero token.
`scripts/parquet_codec_interop.py` compares enabled codecs with Apache Arrow
C++, in both page versions, with dictionaries, nulls and malformed streams.
Checked file/Arrow schema planning requires a root child count, a repetition
type on every non-root node, child counts on groups and no child count on
primitives. The root repetition marker does not add a level. The default
checked schema nesting limit is 128; low-level column-path level lookup is
iterative and does not impose that limit. `scripts/parquet_schema_interop.py`
checks these boundaries using independent Apache Thrift serialization.
For columns whose combined value bytes exceed Arrow's signed 32-bit offset
range, request `ALargeUtf8` or `ALargeBinary` in the target schema passed to
`parquetRowGroupToArrow`. Offset widening works for flat columns and nested
struct/list/map leaves, including indexed selection; it preserves UTF-8 versus
opaque-binary semantics. The ordinary inferred schema keeps its 32-bit storage
and reports an error when that representation cannot hold a whole row group.
The lower-level `Parquet.Read` APIs expose `LargeBinaryBuffers`,
`decodePlainLargeBinaryBuffers`, and the `readLargeBinaryBuffers*` family.
Both widths validate aggregate sizes before copying output payloads.
`scripts/parquet_large_offset_interop.py` checks these projections against
Arrow C++. `scripts/parquet_large_map_interop.py` exercises the official
`large_string_map.brotli.parquet` fixture (two 1 GiB keys), comparing complete
buffer hashes with an independent C++ reader using one-row batches. It needs
several GiB of available memory and is kept separate from the ordinary suite.
`Parquet.File` exposes FILE logical references, subset schemas with field IDs,
and pure resolution plans. Use `NSFile` / `NRFile` with `Parquet.Nested` to
write annotated groups. `parseFileSchema` validates named OPTIONAL children;
`compileFileReader` caches their typed column views for `fileReferenceAt`.
Empty strings retain their raw values, while empty inline bytes remain a valid
source. A present offset (including zero) requires both URI and size. Resolution
prefers inline bytes and retains any external locator as an alternative; it
never applies that locator's range to the inline bytes.
Arrow schema inference preserves FILE identity using arrow-rs's experimental
`arrow.parquet.file` extension (not yet standardized), with empty extension
parameters and parent/child field IDs. `arrowFieldToNestedSchema` and
`columnArrayToNestedRows` lower the extension back into `NSFile` / `NRFile`,
including ordinary, large and view-backed strings/bytes. A plain struct remains
unannotated, and unsupported extension parameters return an error.
`scripts/build_parquet_file_reference.py --output /tmp/parquet-file-reference`
builds an independent reference from a hash-verified upstream source snapshot
and a committed dependency lock (Python 3.12+, Rust/Cargo required; use
`--allow-network` if locked dependencies are not cached). Pass its reported
binary to `scripts/parquet_file_arrow_interop.py --reference <binary>` alongside
the native writer/reader probes. The check covers every nonempty FILE field
subset in all three storage representations, nested lists/structs, IDs, nulls,
and repeated Arrow/Parquet roundtrips.
URI retrieval, relative-URI policy, MIME interpretation and checksum verification
belong to the caller. Recognized checksum tokens are parsed and validated for
encoding; unknown algorithms are ignored. ETAG refers to the whole external
object. The writer rejects inline/locator length disagreements it can prove
locally, without fetching external data.
A zero-field FILE group is retained as a nullable empty struct: it cannot
contain a valid non-null reference, so its rows resolve to null. The writer
supports null FILE rows without inventing dummy fields. Required struct
parents have known presence; optional parents and repeated containers use
sibling leaves to recover their boundaries. Positive-row containers with no
physical leaf to encode their presence/multiplicity are rejected explicitly;
zero-row schemas remain supported. `parquetRowGroupToArrowWithLimits` accepts
`ReadLimits`; the default aggregate budget is 1,048,576 synthesized FILE values.
Callers can raise it for larger valid null-only columns.
`scripts/parquet_file_zero_interop.py` checks independently framed Thrift
footers, native null/anchored-container roundtrips and physical PyArrow values.
It reports upstream limitations explicitly: Rust rejects empty-struct writes
and drops these fields on read; C++ cannot handle several nested empty-group
shapes. PyArrow 26 does not recognize the FILE annotation; `scripts/parquet_file_interop.py` checks it with
independent Apache Thrift and compares the physical values after removing only
that annotation for PyArrow.
Encrypted writes use `Parquet.HighLevel.encodeParquetIO`, which returns
`IO (Either String ByteString)` and generates fresh random module nonces.
Create a new `Parquet.Write.newFooterEncryption` configuration for each file;
share its file identifier and AAD prefix with any encrypted columns. The
supported file mode has an encrypted footer and columns encrypted with the
same footer key, using one file-level AES-GCM or AES-GCM-CTR algorithm.
Separate column keys and signed plaintext footers are rejected. The pure
writer and the mixed nullable writer reject encryption options.
`loadParquetFileEncrypted` authenticates encrypted footers and signed plaintext
footers using the supplied key, file identifier and AAD prefix. Stored identity
fields must agree with the supplied values; an omitted AAD prefix must be
supplied by the caller. `Parquet.Encryption.readFooterCryptoMetadata` exposes
untrusted, opaque key metadata for application-specific key resolution. The
ordinary reader, without a decryption configuration, does not verify signatures.
`ReadOptions.readDecryptionConfig` accepts `Parquet.Decryption.DecryptionConfig`:
explicit footer/column keys or keys resolved by the application from opaque
key metadata, with an optional expected AAD prefix. Schema-path keys use vectors
of path components, so names containing dots stay distinct from nested paths.
Unavailable or incorrect keys for unrequested columns do not block projection;
their statistics remain unavailable. `encryptedColumnKeyMetadata` discovers
authenticated per-column paths and opaque key metadata after opening with a
footer key, without requiring callers to parse Thrift. Requested modules authenticate
under their file, row-group, column and page identities. GCM/CTR data modes,
V1/V2 pages, dictionary pages, nullable/nested columns, encrypted indexes and
separate bloom headers/bitsets use the same file context. Page CRCs cover the
on-disk encrypted bytes and are checked before decrypting the body. CTR page bodies do
not have GCM authentication tags, as defined by the format. The
`scripts/parquet_encryption_interop.py` check reads generated encrypted files
with PyArrow, covering both page versions, both algorithms, selective column
encryption and multiple row groups.
The official encrypted-footer check uses the unmodified C++ library from a
PyArrow 26 wheel, including its mock KMS for the external-key-material fixture:
```sh
python3 scripts/build_parquet_encrypted_reference.py --output /tmp/parquet-encrypted-reference
python3 scripts/parquet_encrypted_read_interop.py \
--reference /tmp/parquet-encrypted-reference \
--reader /path/to/wireform-parquet-reverse-probe \
--corpus /path/to/parquet-testing --output /tmp/encrypted-reads.json
python3 scripts/parquet_encryption_interop.py \
--reference /tmp/parquet-encrypted-reference \
--probe /path/to/wireform-parquet-interop-probe
```
The reader driver compares every physical value and level plus Arrow values,
indexes and bloom filters against C++. It tests explicit and metadata-based
keys, projections without unrelated keys, and page pruning with damaged skipped
and retained headers. The writer driver covers AES-128/192/256, both page
versions and both data ciphers, checking actual column-encryption metadata as
well as values; a successful round trip alone cannot detect an ignored option.
Java fixtures with zero padding inside authenticated metadata are accepted only
after the complete module has been authenticated. The narrower
`parquet_encrypted_footer_interop.py` remains available for footer-only checks.
Floating-point statistics count NaNs separately, omit NaNs from bounds, and
write zero bounds as `-0`/`+0` under `TYPE_ORDER`. Metadata readers preserve
column-order positions; unsupported orders disable that column's min/max
bounds and column index instead of using them for filtering.
Decimal byte-array bounds use signed numeric order across different widths.
Row-group pruning takes `FileMetadata`; column and page checks take a
`StatisticsContext` obtained with `statisticsContext metadata columnIndex`.
This preserves logical decimal and unsigned order and the file's declared
sort order. Modern bounds without a supported sort order are ignored;
compatible legacy signed numeric bounds remain usable. Bloom checks take
the matching `SchemaElement`. Use `PVDecimal` with an exact `Rational` and `PVUnsigned` for those
predicate values. Aggregate min/max use the same logical decoding. Geometry,
geography, interval and unknown logical-type bounds are ignored because their order is undefined.
Readers reject column chunks stored in an external file; load that data file
directly rather than reading its offsets against a summary metadata file.
`LTGeometry` and `LTGeography` carry `GeometryParameters` and
`GeographyParameters`, retaining optional CRS and edge-interpolation fields.
Absent parameters retain the specification's OGC:CRS84/spherical defaults;
explicit defaults and unknown future interpolation codes survive metadata
round trips. `LTFile` preserves a FILE group annotation over its ordinary
inline/locator columns. Applications resolve external FILE URIs.
`FileMetadata.fmKeyValueMetadata` retains ordered `KeyValue` entries,
including duplicate keys, missing values, empty values and opaque bytes.
The Arrow bridge exports geometry/geography as `geoarrow.wkb`, keeping
CRS and edge-interpolation semantics. Inline PROJJSON becomes a JSON object;
`projjson:<key>` references resolve against the file's metadata, and SRIDs
retain their representation hint. No external CRS resources are fetched.
When writing GeoArrow WKB, provide a known CRS: GeoArrow's omitted CRS means
unknown, while Parquet's omitted CRS means `OGC:CRS84`. The conversion returns
an error when it cannot preserve that distinction or an edge algorithm.
Future Parquet edge algorithms remain available in the native footer and
fall back to ordinary binary storage in the Arrow projection.
Page readers verify optional CRC32 checksums over each serialized page body
they read, before decompression. Indexed selection leaves skipped pages and
unused dictionaries untouched. Required selected columns use a dense path;
nullable paths retain definition levels, including nullable ancestors.
Logical annotations and checked widening apply to selected values as well.
Pruning returns whole retained pages; apply a row filter for exact matches.
Byte/text assembly reuses the defined-value buffer and gives null rows empty
offset ranges, including for repeated elements. Null rows never duplicate a
long first value or artificially exhaust the 32-bit offset range.
The header-only parsing functions remain available for encryption framing.
The Arrow bridge preserves modern signed96 TIMESTAMP and legacy INT96 as
distinct lossless extension fields backed by FixedSizeBinary(12). Their full
payloads and timestamp parameters survive schema inference and nested writing.
Conversion to ordinary Arrow timestamps checks range and precision explicitly;
it never wraps or silently drops sub-unit digits.
## Predicate pushdown
`Parquet.Predicate` evaluates a `Columnar.Predicate.Predicate` against:
- per-row-group column statistics from the footer,
- per-page column / offset index entries (Parquet 2.5+),
- split-block bloom filters where available.
The `Wireform.Columnar.decodeFilteredIter` facade entry point in the
umbrella package combines this evaluator with the streaming reader so
callers get only the row groups (and pages, with `PageIndex`) that
the predicate didn't rule out.
`Parquet.Aggregate` short-circuits the entire scan for `count(*)`,
`count(<col>)`, `min(<col>)`, and `max(<col>)` queries when the
required statistics are present in the footer; the reader returns
the aggregate without reading any column data.
Modern min/max bounds must be marked exact for aggregate pushdown; compact
or unmarked bounds remain useful for filtering. Compatible legacy numeric
extrema are also accepted. Empty and all-null row groups are skipped, while
floating extrema require a known zero NaN count. `count(col)` returns
`Nothing` for repeated paths, missing null counts or inconsistent metadata.
`fileRowCountChecked` checks the file total against its row groups without
overflow and returns `Nothing` for invalid counts.
## Arrow bridge
`Parquet.Arrow` converts Parquet column reads to Arrow `ColumnArray`s
and vice versa. The `Wireform.Columnar` facade in `wireform-columnar`
layers a uniform Arrow-shaped API on top of both formats, so a
caller working with Arrow `ColumnArray`s can pick Parquet (or ORC, or
Arrow IPC) by switching the `Format` argument.
The schema-aware writers normalize Arrow millisecond dates to Parquet's
INT32 days, and second-resolution times/timestamps to milliseconds. They
reject fractional-day dates, out-of-day times, timestamp overflow, and
durations without a standard Parquet logical type. Read back using the
inferred Parquet schema to obtain those normalized units.
For nested Arrow columns, pass the same field to
`arrowFieldToNestedSchema field` and `columnArrayToNestedRows field column`,
then use `encodeParquetNested`. This preserves narrow/unsigned integer,
date, time, and timestamp annotations through structs, lists, and maps.
Both Arrow list widths use Parquet's standard LIST layout. Conversion
ignores values beneath null parents and outside referenced child ranges;
visible required nulls and null map keys are rejected. Boolean leaves use
bit-packed PLAIN values. `scripts/parquet_nested_types_interop.py` checks
these paths with PyArrow, DuckDB, and Polars.
Nested files use one V2 page per leaf. The writer checks row/event counts
and the complete page payload against Parquet's signed 32-bit limits.
It bounds cumulative binary/UTF-8/Variant value bytes while shredding and
plans every leaf's level streams before materializing value buffers.
For pre-shredded optional lists, prefer `buildOptionalListFileEither`;
it also checks level ranges, row boundaries, paths, and PLAIN payload sizes.
The older `buildOptionalListFile` raises an error for invalid input.
`Parquet.Footer.writeFooterEither` and `writeRawFooterEither` report footer
length/trailer errors explicitly; the older footer helpers raise errors.
Typed metadata is measured before UTF-8 conversion or output allocation.
Encrypted footers include crypto metadata, the nonce, and authentication
tag in the same checked budget.
The reader preserves fixed byte widths and supports Decimal128/256,
FLOAT16, and fixed binary values through PLAIN, dictionary, and applicable
delta/byte-stream-split encodings. Legacy INT96 timestamps retain their exact
12-byte representation, including dates outside Arrow's Int64 nanosecond range.
Use `parquetFileArrowSchemaChecked` to infer the complete Arrow schema,
then `parquetRowGroupToArrow` to read structs, lists and maps. A target
schema can project/reorder struct children by name, widen `Int32` to `Int64`
and `Float` to `Double` recursively, and request `LargeList` output with
64-bit offsets. List/map wrapper names are matched by position. Unselected
physical leaves are not decoded. A required target rejects visible nulls;
null children hidden beneath null ancestors do not count as visible values.
The reader
preserves null containers, empty containers, null elements and repeated
leaf names under different parents; legacy list and BIT_PACKED level
layouts are supported. Direct physical-leaf reads reject repeated paths.
Legacy timestamp annotations infer UTC, and JSON columns carry Arrow's
canonical `arrow.json` extension metadata.
`scripts/parquet_corpus.py --include-bad-data` also runs Apache's adversarial
fixtures with a per-process heap cap. The malformed Variant fixtures use
unannotated binary metadata/value columns. The corpus check verifies that
ordinary reads preserve those bytes, then explicitly interprets the payloads
through the Variant API and reports `variant-payload-rejected` separately.
### Lossless wide and legacy timestamps
`Parquet.Timestamp` exposes `Timestamp96`, checked integer conversion, modern
little-endian encoding/decoding, and a legacy Julian-day decoder. Modern
`TIMESTAMP` fixed12 storage is signed96 Unix-epoch counts, with independent
millisecond/microsecond/nanosecond and adjusted/local parameters. Legacy INT96
is a distinct signed Julian-day plus signed nanoseconds representation; even
non-normalized nanoseconds remain intact.
The inferred Arrow fields use wireform-specific extensions, not canonical
Apache Arrow extension names. `timestampField` constructs these fields and
`timestampEncoding` validates them. The modern name is
`wireform.parquet.timestamp96`, with metadata `millis:utc`, `millis:local`,
`micros:utc`, `micros:local`, `nanos:utc` or `nanos:local`. Legacy storage uses
`wireform.parquet.int96` with empty metadata. Both use FixedSizeBinary(12);
the original payload bytes are independent of Arrow buffer endianness.
Use `LtTimestamp96 adjusted unit` with `LvFixedBinary` in the nested writer,
or lower an extension field using `arrowFieldToNestedSchema` and
`columnArrayToNestedRows`. `LtInt96` is available for exact legacy copying;
new data should use standard Int64 or modern signed96 TIMESTAMP storage.
`projectTimestampColumn`, row-group schema projection, and page-pruning reads
share checked narrowing. Signed96 can project to an Arrow timestamp with the
same unit and adjusted/local semantics if every present count fits Int64.
Legacy INT96 can project exactly to seconds, milliseconds, microseconds or
nanoseconds. Overflow or discarded sub-unit precision fails. Null payloads
are ignored, including values masked by nullable ancestors. Retaining an
extension requires identical encoding/parameters; removing it explicitly
permits raw fixed12 access.
The independent timestamp suite uses Apache Thrift to adapt only timestamp
schema/type metadata in a temporary copy, then decodes its byte-identical
pages with unmodified PyArrow/C++. This avoids unsupported modern96 semantics
and overflowing legacy conversions in the reference. Full-width Python
integers and published Apache fixtures supply separate semantic checks.
A Spark fixture's extreme legacy payload disagrees with its documented value
and C++ conversion; the original bytes remain exact and overflowing Int64
projection is rejected. This reference discrepancy is reported explicitly.
### Variant binary values and native columns
`Parquet.Variant` implements the v1 metadata/value codec, including all
primitive tags, nested objects/arrays, decimal bounds and IEEE payload bits.
It validates dictionary UTF-8/order flags, key uniqueness/order and each
child's byte boundary. Object values may appear in a different physical
order from their keys. `VariantMetadata` can be compiled or built once and
shared with value-only encoders/decoders; every binary fragment of a shredded
Variant must use that same dictionary.
`Parquet.Variant.Parquet` provides `buildVariantParquetFile`,
`readVariantColumn`, and a compiled unshredded `VariantReader`. File reads
reconstruct both unshredded and typed Variant groups, including recursive
objects, arrays and partial objects. They access fields by name and distinguish
a missing column slot from a present Variant null. Native `NSVariant` writes now require valid payloads and emit
the `VARIANT(1)` logical annotation. Ordinary Arrow conversion continues to
expose the physical storage; use the explicit semantic API to interpret it.
`VariantLimits` bounds nesting depth, total tree values and dictionary entries.
Defaults are 500 levels and 1,048,576 values/entries. The `WithLimits` helpers
and `NSVariantWithLimits` allow callers to raise those bounds for larger valid
data. The default depth follows Apache's adversarial corpus policy.
`scripts/parquet_variant_interop.py` compares native unshredded files with
PyArrow physical values, Apache Thrift annotations, independently annotated
foreign files, and pinned upstream Rust binary semantics. Build that oracle
with `scripts/build_parquet_variant_reference.py` (Python 3.12+, Cargo and
the locked dependency cache; `--allow-network` can fetch missing dependencies).
The driver records reference limitations rather than treating unsupported
cases as successes.
`Parquet.Variant.Schema` checks canonical `arrow.parquet.variant` storage
fields, including exact primitive mappings, decimal precision bands,
non-null object/list wrappers, and dictionary/run-end encoded metadata.
Schema validation alone does not reconstruct a shredded value.
The nested writer supports `LtDecimal precision scale` through precision 76,
using INT32, INT64, fixed 16-byte, or fixed 32-byte storage according to the
precision. Fixed-byte values use `LvFixedBinary` with `LtFixedBinary` or
`LtUuid`; they are emitted without a BYTE_ARRAY length prefix. The Arrow
nested helpers retain decimal precision/scale and UUID annotations and ignore
poisoned child storage beneath null parents.
`Parquet.Variant.Arrow` reconstructs canonical Arrow Variant columns with
`compileVariantArrowReader` and `variantArrowAt`. It covers every canonical
primitive, recursive objects/arrays, partial objects and value-only/typed-only
wrappers. Metadata may use binary/large/view storage, dictionary keys or run-end
encoding; encoded metadata is resolved by logical index without expansion.
One value/depth budget covers typed nodes and all fallback fragments. Missing
object fields, present Variant nulls and null root slots remain distinct.
`scripts/parquet_variant_typed_interop.py` verifies raw encoded semantics and
compares with the pinned arrow-rs reconstruction kernel. It explicitly checks
reference date/timestamp range limits and nanosecond tag narrowing, while
requiring Haskell to preserve the complete raw domain and exact Variant type.
Build that oracle with `build_parquet_variant_reference.py --typed`.
Native `VARIANT(1)` inference preserves the canonical Arrow extension and exact
primitive identity, including Decimal32/64, nanosecond timestamps and UUIDs.
Typed layouts enforce the published physical types and repetition rules;
retaining the Variant extension during projection requires the complete,
unchanged semantic storage tree. Field reordering is allowed. Remove the
extension explicitly for raw physical projections that drop or widen fields.
Optional zero-leaf typed objects have no observable presence for nonempty
native files and are rejected; zero-row schemas remain representable.
`scripts/parquet_variant_native_interop.py` lowers the shared typed fixtures
into foreign-written Parquet pages and checks native reads plus Rust
reconstruction from the inferred Arrow schema. It reports metadata storage
normalization and the two empty-object cases' unshredded representation.
Malformed native carriers, repetitions and list wrappers are rejected even
when the file has no rows. `scripts/parquet_narrow_integer_interop.py` checks
signed/unsigned 8/16-bit boundaries in flat and nested, required and optional
columns; out-of-domain physical values fail instead of wrapping.
`compileVariantStorageSchema` compiles a canonical Arrow Variant field into
an opaque schema for `NSVariantTyped`. `columnArrayToNestedRows` and the
nested writer preserve typed storage, the native annotation, all field IDs,
and the exact original metadata and supplied fallback bytes. Named Variant
children bind before validation and row lowering, so their in-memory order
may differ from the field. Binary views,
dictionary/run-end encoded metadata and list views are resolved by logical
index. Typed-only wrappers gain a native binary fallback column; missing
required root/array values become explicit Variant nulls, while missing object
fields stay absent. Required children are checked only below present parents.
All native leaf byte/count preflights precede semantic fragment decoding.
`Parquet.Variant.Shred` builds native rows from semantic Variant values.
Compile a declared shape with `compileVariantShredder`, use
`shredVariantWithMetadata` to retain an existing dictionary (including fully
shredded names), and pass `variantShredSchema` plus the returned rows to
`buildNestedFile`. `shredVariant` creates one complete dictionary per root
value when no original dictionary exists. Every fallback fragment shares it.
A primitive enters typed storage only when its Variant constructor matches
exactly. Decimals also require the same scale and a fitting coefficient;
integer width, timestamp unit/time zone and decimal width/scale mismatches
keep their original binary tag and value. No rescaling or widening is done.
`scripts/parquet_variant_write_interop.py` checks both physical storage
forwarding and semantic shredding, with independent raw-value expectations,
PyArrow storage checks and the pinned Rust reconstruction kernel. Supplied
metadata/fallback blobs are checked byte for byte in the forwarding path.
PyArrow's zero-row empty-group rejection is an explicitly asserted reference
limitation; nonempty unobservable typed layouts are rejected by the writer.
The remaining full-format release checks and final comparative throughput /
allocation measurements are still open. These APIs alone do not establish
full Parquet conformance or performance parity.
## Benchmarks
The file throughput and indexed-selection comparisons are declared in
`scripts/bench-manifest.json` at the repository root. Run them sequentially:
```bash
python3 scripts/run-benchmarks.py --only parquet-throughput
python3 scripts/run-benchmarks.py --only selection-throughput
```
The whole-file harness checks all 400,000 values, schema and page layout
before timing. Reads return fully decoded Arrow batches, and Boolean inputs
are forced before the clock starts. The three writers use required PLAIN V2
columns and one 100,000-row page per column; the C++ comparison explicitly
sets both `column_encoding="PLAIN"` and `max_rows_per_page=100000` because
its defaults differ. The Python-bound C++ writer also emits page statistics;
wireform and the configured Rust writer emit chunk statistics.
`scripts/parquet_file_preflight.py` takes `--wireform-bench` and `--rust-bench`
paths to the built benchmark binaries. It verifies nine independently written
files and nine rejection controls, including last-row corruption and different
page/group layouts. The Haskell benchmark also provides `--verify-fixtures DIR`;
the Rust benchmark provides `--write-fixtures DIR`. Pass `--wireform-bench` to
`scripts/parquet_bench_compare.py` to verify C++ output layout before timing.
The selection comparison requires PyArrow and Apache Thrift in the Python
environment. Its fixture generator creates six multipage integer/text cases
with required/nullable values, plain/dictionary encoding, and three codecs.
Haskell and parquet-rs read identical bytes, evaluate the same page maximum
predicate, and return whole retained pages. Each harness checks every returned
value against independently sliced PyArrow output before timing. Timings
include footer/index parsing and materialization; file IO and fixture generation
are excluded. Haskell also records allocated bytes. These workloads complement
the flat required file baseline; they do not establish performance for every
supported type or generated record helper.
A criterion harness in [`bench/Bench.hs`](bench/Bench.hs) measures
the C-vs-pure speedup on the XXH64 hash that Parquet bloom filters
key off:
```bash
cabal bench wireform-parquet:parquet-bench
```
The recorded XXH64 measurements and chart sources remain in
[`bench-results/summary/parquet-xxh64-c-vs-pure.json`](https://github.com/iand675/wireform-/blob/main/wireform-parquet/bench-results/summary/parquet-xxh64-c-vs-pure.json).
They describe the toolchain and capture date recorded in that file; they are
not a whole-library performance claim. Charts and tables are generated by
`regen-stats`; do not edit their measured values by hand.
For cross-language comparisons (file-format-level):
- Haskell:
[`parquet-hs`](https://hackage.haskell.org/package/parquet-hs) (the
established Hackage Parquet library) and
[`parquet`](https://hackage.haskell.org/package/parquet).
- C++: [arrow-cpp](https://github.com/apache/arrow/tree/main/cpp/src/parquet),
the de facto reference. DuckDB also ships a high-quality reader.
- Rust: [`parquet`](https://crates.io/crates/parquet), used by
Datafusion and Polars.
- Python: [`pyarrow.parquet`](https://arrow.apache.org/docs/python/parquet.html).
## Specification references
- [Apache Parquet specification](https://github.com/apache/parquet-format)
- [Parquet Thrift schema](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
- [Modular encryption](https://github.com/apache/parquet-format/blob/master/Encryption.md)
- [Bloom filters](https://github.com/apache/parquet-format/blob/master/BloomFilter.md)
## Nested writer options
`encodeParquetNested` preserves the original uncompressed V2 layout.
`encodeParquetNestedWith` selects compression, V1/V2 pages and generated
indexes through `WriteOptions`. The lower-level `buildNestedFileWithAux`
also accepts pre-built bloom filters and explicit single-page indexes per
physical leaf. Repetition/definition streams are compressed with values in
V1; V2 compresses values only. Metadata sizes include the actual serialized
header. Unsupported encryption and name-based bloom options are rejected.
Automatic min/max bounds for non-null nested leaves remain open; all-null
column indexes and offset indexes are generated now.
Run the independent full-codec matrix with all optional codecs enabled:
```sh
python3 scripts/parquet_nested_options_interop.py --probe /path/to/writer-probe \
--lzo-reference /path/to/columnar-lzo-throughput --output-dir /tmp/nested-check \
--aux-fixtures /path/to/unit-fixtures --reference /path/to/encrypted-reference
```
The LZO reference is the `columnar-lzo-throughput` executable linked to
unmodified liblzo2. It converts compression only before independent value
reads; page versions, sizes and offset-index entries are checked on the
original file through Apache Thrift. PyArrow, DuckDB and Polars compare all
logical values/types against independent Arrow models. The old nested type
catalogue additionally checks byte-for-byte preservation of the legacy path.
## Generated nested bloom filters
`encodeParquetNestedWith` accepts `writeBloomFilters`. Each name identifies a
literal top-level field (including names containing dots), and selects every
physical leaf below that field. Unknown names return `Left`; nulls and empty
containers contribute no values. `buildNestedFileWithBloomFilters` combines
this selection with explicit per-leaf auxiliary options; supplied filters
win over generated filters.
Construction hashes PLAIN values with XXH64 seed zero. Byte-array length
prefixes are excluded, and Boolean page bits are expanded to individual 0/1
bytes for hashing. V2 compression and bloom construction share a materialized
values buffer. Unrequested leaves retain their existing direct builder path.
The independent checker uses unmodified Apache Arrow C++ to decode every
present value and probe its filter. LZO files additionally use unmodified
liblzo2 for compression-only conversion; their original bloom bytes must
match the corresponding uncompressed control.
```sh
python3 scripts/parquet_nested_bloom_interop.py \
--fixtures /path/to/regression-fixtures --reference /path/to/reference \
--lzo-reference /path/to/columnar-lzo-throughput --output /tmp/bloom-check.json
```
Bloom throughput is a deferred follow-up. This functionality does not imply
completion of nested min/max generation or the remaining encrypted writers.
## Nested statistics and explicit orderings
The writer with auxiliary options generates column-chunk statistics even when
page indexes are disabled. `caGeneratePageIndex` controls the additional
column/offset indexes. The legacy no-options entry point retains its original
bytes. PLAIN values are materialized once and shared with bloom generation
and V2 compression; the bounds scan retains offsets without decoded vectors
or per-value slices.
Bounds use the actual emitted schema, including unsigned integers, wide
signed decimals, UUIDs and signed96 little-endian timestamps. Floating-point
columns use `IEEE_754_TOTAL_ORDER`: NaNs are counted, bounds exclude them when
numeric values exist, and NaN-only pages contain their IEEE-ordered NaN
extrema. Signed zero and NaN payload bits are preserved. FLOAT16 uses the
native `LtFloat16` / `LvFloat16 Word16` surface and the nested Arrow bridge.
Legacy INT96 uses `INT96_TIMESTAMP_ORDER`, comparing the signed Julian-day
part first and the signed nanoseconds part second. This differs from signed96
Unix timestamp ordering. Types without a defined logical order get no
ordered bounds.
`scripts/parquet_nested_statistics_interop.py` checks the literal footer and
page-index fields through Apache Thrift, recomputes extrema using independent
Python scalar models, and checks values/bloom membership with unmodified
Arrow C++. Its binary16 bridge checks retain raw signed-zero bits through
PyArrow. LZO conversion changes compression only, through unmodified liblzo2;
original indexes and bloom regions are checked before conversion.
## Encrypted nullable/nested writing
The IO writers share the existing pure encoders and compress before encrypting
page bodies. The required-column/footer-key path retains its existing direct
builder. The new private `Parquet.Write.Encrypted` assembly seam supports the
single-data-page layout produced by the mixed and nested writers; it is not a
public rewriter for arbitrary third-party files. It relocates page, bloom and
index regions, rebuilds offset indexes, and measures metadata before encoding.
Column keys use their actual physical schema paths. Separate-key metadata is
encrypted; signed plaintext footers retain only statistics-free legacy column
metadata. Both GCM and GCM/CTR authenticate headers, metadata, indexes and blooms.
Every module gets a fresh nonce and physical row-group/column/page AAD.
`parquet_encrypted_writer_interop.py` requires Python `cryptography` alongside
the existing PyArrow26/Apache Thrift dependencies. It authenticates/decrypts
all module types independently, compares exact compressed bytes and metadata
to plaintext controls, rejects repeated key/nonce pairs, and reads non-LZO
files directly through unmodified Arrow C++. For LZO it checks every original
encrypted module, then uses unmodified liblzo2 and C++ on plaintext controls.
The grid covers 672files: two layouts, seven codecs, three AES widths, two
algorithms, two footer modes, two key routes and two page versions. Native
regressions also check tampering, missing/wrong keys and public projections.
## Encrypted writer benchmarks
Run the matched whole-file comparison sequentially from the repository root:
```sh
python3 scripts/run-benchmarks.py --only encryption-throughput
```
The native Criterion target is `parquet-encryption-throughput`; its independent
reference is the pinned parquet-rs harness in `interop/parquet-encryption-reference`.
Both write nullable scalar or optional-list/optional-element INT64 columns at
64 and 8192 rows, with separate column keys, 128-bit AES-GCM, stored AAD prefixes,
V2 pages, page statistics and both indexes. They use uncompressed single-page
columns and exercise encrypted and signed plaintext footers. Input construction
is outside the timed region; encryption configuration and fresh randomness are
included. Bloom performance remains a separate follow-up.
The nested native API validates dynamically shaped `NestedRow` values before
allocating exact-size buffers. Its reference starts with typed, validated Arrow
arrays. Keep that semantic difference explicit when interpreting whole-file
timings. Inspect Core and allocation regressions alongside elapsed time: the
row/element cursors and fixed-width count loop should stay unboxed, and no
per-value temporary list or validation result should be allocated. Large accepted
plans must retain lazy payloads, as exercised by the size-boundary regressions.
The writer uses crypton's cached AES-GCM key context and system RNG. Keep the
`crypton` and `ram` bounds consistent in the workspace and Nix package sets when
upgrading them; reader/writer interoperability and existing authentication/hash
vectors must pass against the same dependency versions.