Skip to content

Migration 0.17 → 0.18

Status: required for consumers that persist plans or cross dataframe-engine boundaries.

Summary

ETLantic 0.18.0 adds Gate A versioned tabular interchange. Compatible Polars↔Pandas boundaries can now record and execute an etlantic.interchange/1 descriptor with explicit mechanism, ownership, collection, fidelity, and evidence bounds.

Install

pip install -U "etlantic[dataframes,arrow]==0.18.0"

Official plugins require etlantic>=0.18.0,<0.19. Third-party plugins should publish a matching minor release after validating their capability declarations and interchange behavior.

Regenerate stored plans

Plans generated by 0.17 do not contain versioned interchange descriptors and are not silently rewritten. Regenerate and review them:

etlantic validate path/to/pipeline.py:Pipeline --profile profile.json --format json
etlantic plan path/to/pipeline.py:Pipeline --profile profile.json --format json
etlantic plan explain path/to/pipeline.py:Pipeline --profile profile.json --format json

Confirm every new etlantic.interchange/1 descriptor has the expected producer and consumer engines, mechanism, ownership, collection policy, copy eligibility, and fidelity limits. Invented or invalid descriptors fail closed.

Arrow selection and fallback

When both plugins advertise compatible Arrow capabilities and the declared bounds are satisfied, Gate A selects the highest-priority safe Arrow mechanism. Selection can instead choose an explicit records or native fallback when Arrow is unavailable or contract-unsafe.

The pre-0.18 Arrow-assisted conversion helper remains a legacy best-effort path. It does not provide a versioned plan decision or Gate A evidence. Do not treat successful best-effort conversion as proof that a Gate A boundary was planned or honored.

Scope

  • Gate A conformance covers Polars↔Pandas.
  • PySpark and SQL Arrow physical boundaries remain follow-up work.
  • DataFusion did not ship; it remains planned for non-blocking Gate B / 0.19+.
  • PyArrow remains optional and outside the core dependency set.

Checklist

  • Upgrade core and all official plugins to 0.18.0
  • Regenerate persisted 0.17 plans
  • Review interchange descriptors and fallback reasons
  • Re-run plugin and cross-engine conformance tests
  • Update production plugin allowlist pins to the 0.18 plugin minor