Engine selection¶

Status: Available in ETLantic 0.53.0 (Beta release candidate).

PyPI vs clone

Recommended path below is PyPI-only. Rows marked clone-assisted need a git checkout and uv sync.

Recommended next step¶

After Quickstart succeeds on local Python with its generated portable transformation:

python -m pip install 'etlantic[polars]==0.53.0'
Set dataframe_engine to "polars"; keep portable_transform_policy="require" and the transformation body unchanged as shown in the Polars tutorial (PyPI path)
Re-run validate → plan → run on the same project

Stay on local until that path is green. Add SQL or Spark only after one dataframe engine works.

Goal	Install	Profile hint	Guide
Learn the model with JSON files	`etlantic==0.53.0`	`development`	Quickstart
JSON / CSV files	core only	file storage bindings	File storage (PyPI)
Fast local dataframes	`etlantic[polars]==0.53.0`	`dataframe_engine="polars"`, portable policy `"require"`	Polars tutorial (PyPI path)
Pandas compatibility	`etlantic[pandas]==0.53.0`	`dataframe_engine="pandas"`, portable policy `"require"`	Pandas tutorial (PyPI path)
Cross-engine Polars↔Pandas	`etlantic[dataframes]==0.53.0`	both plugins allowlisted	Interchange example (clone)
Keep work in SQL	`etlantic[sql]==0.53.0`	`sql_engine="sql"`	SQL hello (PyPI) → SQL tutorial (clone)
Local Spark batch	`etlantic[pyspark]==0.53.0`	`spark_engine="pyspark"` (needs Java)	PySpark tutorial (clone-assisted)
Emit Airflow DAGs	`etlantic[airflow]==0.53.0`	`orchestrator="airflow"`	Airflow tutorial
Prefect local scheduler	`etlantic[prefect]==0.53.0`	`orchestrator="prefect"`	Prefect example (clone)
Portable transforms (recommended)	matching engine plugin	`portable_transform_policy="require"`	Portable transforms

One engine first. Do not combine SQL + Spark + dataframes until a single engine path works under validate and plan.
Pin the minor in 0.x. Keep core and every official plugin on the same release (for example etlantic==0.53.0 with etlantic-polars==0.53.0).
Production profiles need allowlists. Create profiles/prod.json from the embedded JSON in Capabilities → CI starter. Trim the allowlist to engines you install.
Airflow is compile-only. etlantic-airflow writes DAG artifacts; install Apache Airflow separately where DAGs load.
Memory demos need Python seeding. CLI run does not share process-local memory from a previous Python session.
Author logic once. Use @Transformation.portable within the qualified baseline. Native bodies pin a step to an engine and are not eligible for adaptive execution.