Skip to content

Run a Local PySpark Batch

Status: Available in ETLantic 0.15.0. Structured Streaming remains experimental.

Repository examples

Companion scripts under examples/ are not installed with the PyPI wheel. Clone a matching checkout (prefer the v0.15.0 tag) and use uv sync / the documented dependency group before running them.

Prerequisites

  • Python 3.11+
  • A Java runtime supported by your PySpark installation
  • etlantic-pyspark==0.15.0
python -m pip install 'etlantic==0.15.0' 'etlantic-pyspark==0.15.0'
git clone https://github.com/eddiethedean/etlantic.git
cd etlantic
python examples/pyspark_local.py

The transformation registers a native PySpark implementation and selects it with Profile(name="spark-local", spark_engine="pyspark"). The runtime also registers the local Spark provider explicitly.

Complete source: examples/pyspark_local.py.

Managed Databricks, EMR, and Spark Connect providers are not included. See PySpark execution and compatibility.