Skip to content
Typed, contract-driven pipelines

Design once.
Validate everywhere.

Model data pipelines in Python, validate them as contracts, and run them where your engines already are.

Quickstart Capabilities

ETLantic is a Python framework for defining typed, contract-driven data pipelines and coordinating their execution through the tools users already choose.

Its central idea is simple:

Define data, transformations, and pipelines with typed Python classes. Validate and plan them once. Execute them through interchangeable backends.

ETLantic is inspired by FastAPI's type-driven developer experience, but it does not turn ETL into a web API metaphor. It applies the same principle—types as executable interface declarations—to data engineering.

Project Status

The published 0.15.0 alpha ships validation, profiles, an immutable secret-free PipelinePlan, local Python execution, runtime secret resolution, run reports, memory/callable/JSON/CSV storage, a versioned dataframe protocol with Polars and Pandas plugins, a versioned SQL protocol with the etlantic-sql PostgreSQL reference plugin, a versioned Spark protocol with the etlantic-pyspark reference plugin (local provider + portable compiler), and a versioned orchestration protocol with the etlantic-airflow reference compiler. Portable Polars and PySpark compilers claim kernel + portable-relational/1; the Pandas eager compiler ships with the same claims in 0.14. Structured Streaming APIs are experimental.

Green path (start here only)

  1. What's new in 0.14 — adopter delta
  2. Installation — pip install etlantic==0.15.0
  3. Quickstart — five-minute success
  4. Capabilities — shipped vs not
  5. Evaluator brief — for decision-makers
  6. Pilot walkthrough — controlled pilot

Pages marked Future design are not APIs. Capabilities is the single source of truth. Prefer the Green path above; persona paths below are optional after first success.

Read Available pages and the Green path first. Chapters under Design Proposals and Examples marked Future design are not current APIs. Design studies are not a promise that every illustrated surface is installable.

Portable PySpark-inspired authoring ships in 0.11 via @Transformation.portable and etlantic.transform, emitting dtcs.transform-plan/2. 0.12 added planning integration and Polars kernel portable execution. 0.13 shipped Polars and PySpark portable-relational/1 compilers; 0.14 shipped the Pandas eager compiler and public conformance SDK. Safe SQL portable lowering for that claim set shipped in 0.15; advanced profiles follow as 0.15 continuation work. Start with Portable Transformations.

Minimal working example

from etlantic import (
    Data,
    Input,
    Output,
    Pipeline,
    PipelineRuntime,
    Load,
    Extract,
    Transformation,
)


class RawCustomer(Data):
    customer_id: int
    first_name: str
    last_name: str


class Customer(Data):
    customer_id: int
    full_name: str


class NormalizeCustomers(Transformation):
    customers: Input[RawCustomer]
    result: Output[Customer]


@NormalizeCustomers.implementation("local")
def normalize_customers(customers: list[RawCustomer]) -> list[Customer]:
    return [
        Customer(
            customer_id=row.customer_id,
            full_name=f"{row.first_name} {row.last_name}",
        )
        for row in customers
    ]


class CustomerPipeline(Pipeline):
    raw: Extract[RawCustomer] = Extract(asset="customer_source")
    normalized = NormalizeCustomers.step(customers=raw)
    curated: Load[Customer] = Load(
        input=normalized.result,
        asset="customer_sink",
    )


def main() -> None:
    CustomerPipeline.validate(profile="development").raise_for_errors()
    runtime = PipelineRuntime()
    runtime.memory.seed(
        "customer_source",
        [RawCustomer(customer_id=1, first_name="Ada", last_name="Lovelace")],
    )
    CustomerPipeline.run(profile="development", runtime=runtime)
    print(runtime.memory.get("customer_sink"))


if __name__ == "__main__":
    main()

Copy into a file and run with Python after pip install etlantic, or use examples/quickstart.py. From these declarations ETLantic can also generate ODCS / DTCS / DPCS contracts, Mermaid lineage, and a secret-free PipelinePlan. Optional plugins add Polars, Pandas, SQL, PySpark, and Airflow compilation.

The transformation implementation remains separate—register "polars", "sql", and other engines the same way after installing the matching plugin.

The Architecture in One View

Typed Python authoring or portable contracts
                    │
                    ▼
          Typed logical pipeline model
                    │
                    ▼
     Introspection and semantic validation
                    │
                    ▼
        Profile and capability resolution
                    │
                    ▼
        Immutable PipelinePlan (resolved IR)
                    │
          ┌─────────┼──────────┐
          ▼         ▼          ▼
      Execute    Compile    Generate
          │         │          │
          ▼         ▼          ▼
      Plugins   Airflow/SQL  Docs/graphs

ETLantic owns modeling, validation, planning, and coordination.

Standards own contract meaning. ContractModel owns data-contract operationalization. Plugins and external systems perform the work.

The Three Contract Authorities

ETLantic intentionally recognizes only three top-level contract families:

Contract Authority Answers
Data contract ODCS and ContractModel What is valid data?
Transformation contract DTCS How is data expected to change?
Pipeline contract DPCS How are data and transformations composed?

Profiles, plugins, resources, callbacks, artifacts, and execution plans are implementation or runtime concepts—not additional contract standards.

Choose Your Path

Follow the Green path above for first success. Optional persona forks:

I want to run something in five minutes

Same as the Green path: Installation → Quickstart → then Capabilities. Runnable code: examples/quickstart.py, examples/file_storage.py.

I want to understand the idea

  1. Manifesto
  2. Evaluator brief
  3. Core Concepts
  4. Architecture
  5. Documentation Status

I want to author pipelines

  1. Getting Started
  2. Data Contracts
  3. Transformations
  4. Pipelines

I want to understand execution (shipped)

  1. Execution Model
  2. Local Python
  3. Secrets Management (env + file only)
  4. Polars / Pandas
  5. SQL
  6. Run Reports

I want runnable examples

I want design studies (aspirational)

These pages describe intended 1.0 workflows. They are not current API guides:

I want to build a dataframe or SQL plugin

  1. Dataframe Plugin protocol
  2. SQL Plugin protocol
  3. Dataframe Plugins overview
  4. SQL overview
  5. Compatibility

I want other plugins (future)

  1. Plugin SDK overview (future design)
  2. Testing Plugins

I am integrating or migrating SparkForge

  1. SparkForge Feature Adoption
  2. Roadmap

Documentation Map

Section Purpose
Getting Started Learn the core workflow
Foundations Understand product philosophy and architecture
Data Contracts Define and operationalize typed datasets
Transformations Define typed transformation interfaces
Pipelines Compose transformations into portable graphs
Execution Local runtime, secrets, dataframe and SQL engines
Orchestration Airflow compiler (etlantic-airflow); other orchestrators future
Visualization Mermaid, Graphviz DOT, HTML lineage
Examples Runnable pointers + design studies
Reference CLI, API, compatibility
Development Contributing, roadmap, release
Specifications Normative DTCS and DPCS documents

Non-Goals

ETLantic is not intended to become:

  • A dataframe engine
  • A distributed scheduler
  • A storage system
  • A secret manager
  • A proprietary pipeline contract format
  • A replacement for Pandas, Polars, SQL engines, Spark, Airflow, or Dagster

It is the typed control and interoperability layer that connects those systems without allowing any one of them to define the pipeline's portable meaning.