SQL¶

SQL plugins execute eligible transformations inside a database while preserving logical semantics from DTCS and the Pipeline Plan.

Status: shipped in 0.6.0 via the etlantic-sql PostgreSQL reference plugin. SQLite is supported for local demos only.

Safe portable SQL lowering for kernel + portable-relational/1 ships in 0.15 via the etlantic-sql transform compiler. Native @implementation("sql") remains fully supported.

ETLantic does not depend on database drivers. Install the plugin separately:

pip install etlantic-sql
export ETLANTIC_SQL_URL=postgresql+psycopg://user:pass@localhost:5432/etlantic

Protocol¶

The versioned protocol is etlantic.sql/1. Plugins compile typed expressions and write intents, execute against relations, and report capabilities. The local orchestrator consumes the resolved PipelinePlan without reselecting an engine.

Profile and implementations¶

from etlantic import Profile
from etlantic.sql import RelationRef, col, concat, select

Profile(name="sql-prod", sql_engine="sql")

@NormalizeCustomers.implementation("sql")
def normalize_sql(customers: RelationRef):
    return select(
        col("customer_id"),
        concat(col("first_name"), col("last_name"), as_="full_name"),
        source=customers,
    )

Select the engine with Profile.sql_engine = "sql". Plugins are discovered through the etlantic.sql_plugins entry point.

Portable SQL lowering (shipped in 0.15)¶

Portable kernel and portable-relational/1 nodes lower into typed etlantic.sql/1 IR and then into dialect SQL (PostgreSQL via etlantic-sql is the reference):

dtcs.transform-plan/2
        ↓
SqlQuery / typed SQL expressions
        ↓
safe dialect compiler

from etlantic.testing import run_portable_transform_conformance_suite
from etlantic_sql import create_transform_compiler

run_portable_transform_conformance_suite(create_transform_compiler())

Identifiers remain validated and values remain bound parameters. Portable definitions cannot contain trusted SQL fragments or raw F.expr() strings. Unsupported dialect functions, types, null behavior, or ordering fail during planning with PMXFORM* diagnostics — never via raw SQL or UDF approximation.

Policy when portable SQL cannot claim the needed profile:

portable_transform_policy="require" fails closed at planning;
prefer may select an explicit native @implementation("sql") only — never silent portable emulation, and never an implicit fall-back to Polars or another dataframe engine;
native prefers a registered SQL implementation.

Advanced families (window, complex types/values, reshape, …) remain under the 0.15 continuation backlog.

SQL→SQL without Python fetch¶

When adjacent SQL steps and sinks share a database, ETLantic fuses execution so intermediate rows are not materialized in Python.

Capabilities¶

Plugins publish capabilities such as transactions, catalog inspection, and atomic rename/swap. The 0.6 etlantic-sql reference plugin does not advertise MERGE (sql_merge=False); requiring merge fails closed at planning. Unsupported requirements fail at validation or planning (fail closed).