Portable Customer Transformation

Status: Available in ETLantic 0.15 (Polars + PySpark + Pandas + SQL relational)

@Transformation.portable authoring shipped in 0.11. This guide runs a kernel or relational plan on Polars/PySpark/Pandas without a native @implementation(...) for the advertised claim set. Safe SQL portable lowering for that claim set also shipped in 0.15 (etlantic-sql).

Runnable companion: examples/portable_polars_kernel.py.

pip install 'etlantic==0.15.0' 'etlantic-polars==0.15.0'
python examples/portable_polars_kernel.py

Author once

from etlantic import (
    Data,
    Input,
    Output,
    Parameter,
    Pipeline,
    PipelineRuntime,
    Profile,
    Load,
    Extract,
    Transformation,
)
from etlantic.plan import explain_plan, plan_pipeline
from etlantic.registry import PlanningContext
from etlantic.transform import functions as F
from etlantic_polars import create_plugin


class RawCustomer(Data):
    customer_id: int
    email: str
    age: int


class Customer(Data):
    customer_id: int
    email: str
    age: int


class NormalizeCustomers(Transformation):
    customers: Input[RawCustomer]
    minimum_age: Parameter[int] = 18
    result: Output[Customer]


@NormalizeCustomers.portable
def normalize(customers, minimum_age):
    return (
        customers.filter(F.col("age") >= minimum_age)
        .withColumn("email", F.lower(F.col("email")))
        .select("customer_id", "email", "age")
    )


class PortablePolarsPipeline(Pipeline):
    raw: Extract[RawCustomer] = Extract(asset="customers")
    normalized = NormalizeCustomers.step(customers=raw)
    curated: Load[Customer] = Load(input=normalized.result, asset="curated")

Inspect the emitted plan (no engine required):

plan = NormalizeCustomers.to_transform_plan()
assert plan["planIdentity"] == "dtcs.transform-plan/2"
print(NormalizeCustomers.portable_fingerprint())

Run on Polars without a native callable

profile = Profile(
    name="polars-portable",
    dataframe_engine="polars",
    portable_transform_policy="require",  # fail closed if unsupported
)
runtime = PipelineRuntime()
runtime.register_dataframe_plugin("polars", create_plugin())
runtime.memory.seed(
    "customers",
    [
        RawCustomer(customer_id=1, email="A@X.COM", age=30),
        RawCustomer(customer_id=2, email="b@y.com", age=10),
    ],
)
context = PlanningContext.create(profile=profile, registry=runtime.registry)
planned = plan_pipeline(PortablePolarsPipeline, context=context)
assert planned.implementations["normalized"].kind == "portable_compiled"
report = PortablePolarsPipeline.run(
    profile=profile, runtime=runtime, context=context
)
print(runtime.memory.get("curated"))

Default policy is prefer (portable when covered; diagnosed native fallback otherwise). Use native to ignore compilers.

Expected plan evidence

{
  "node": "normalized",
  "implementation_kind": "portable_compiled",
  "compiler": {"name": "etlantic-polars"},
  "ir_fingerprint": "<64-char sha256>"
}

Use explain_plan(planned) or etlantic plan … --format json.

Expected outputs

customer_id	email	age
1	a@x.com	30

Age 10 is filtered out. Email is lowercased.

Unsupported operations fail closed

Joins, windows, and conversion-profile functions such as dtcs:cast are outside the 0.12 Polars kernel claim. With portable_transform_policy="require", planning raises PipelineValidationError with PMXFORM301 (or PMXFORM302 when no compiler is discoverable).

What remains future

• Safe SQL portable lowering for kernel + portable-relational/1 (0.15 exit gate)
• Advanced portable profile graduation (window, reshape, …) under the 0.15 continuation backlog after the SQL gate

Polars, PySpark, and Pandas relational compilers already ship in 0.13–0.14. Keep @implementation("sql") until the portable SQL compiler ships; keep native callables for profiles outside the advertised claim set.