Frequently Asked Questions¶
Status: Available in ETLantic 0.53.0 (Beta release candidate).
Practical questions for ETLantic 0.53.0. Philosophy and contract taxonomy live under Foundations.
What is ETLantic?¶
A typed Python framework for contract-driven data pipelines. You model once,
validate before write, then run or compile on local Python, Polars, Pandas,
SQL, or Spark. Optional etlantic schedule / scheduler serve / worker serve
add a timer service wrapping CP3 durable work; FastAPI never executes
pipelines. It is not dbt and not a dataframe engine. See
Compare for when not to use it.
What is a Profile?¶
A Profile selects engines, asset bindings, trust mode, and (optionally)
observability. Prefer named profiles (development, production) and pass
--profile on validate / plan / run. Fail-closed production trust uses
security_mode="production" plus a non-empty plugin_allowlist — not the
profile name alone. Hub: Profiles.
Why file:Class / path.py:ClassName?¶
CLI targets load a Python module then a class:
pipeline.py:SamplePipeline. That is a filesystem path plus a class name,
not a URL scheme. See CLI — Pipeline targets.
Where do reports go?¶
By default under the workspace .etlantic/reports/ (and history under
.etlantic/history/ when configured). Use etlantic report list /
etlantic report query. Hub: Reports and history.
Do I need Java?¶
Only for PySpark. Local, Polars, Pandas, and SQL (SQLite/PostgreSQL) do not require a JVM. See Engine selection.
Which engine should I start with?¶
Built-in local Python (Quickstart). Then Polars for a first dataframe
engine. SQL needs etlantic-sql (+ PostgreSQL for MERGE). PySpark needs Java.
Must core and plugin versions match?¶
Yes. Pin the same minor:
Why do validate/plan work but run has no data?¶
Validate/plan do not need source rows. Quickstart binds JSON under data/ —
check files exist and you use the same --profile for validate/plan/run.
Profile name vs security_mode?¶
Production fail-closed trust keys off security_mode="production" and a
non-empty plugin_allowlist — not the profile name. See
Profiles hub.
How do I pass secrets?¶
Use SecretRef — never put values in plans. Follow the
Secrets decision tree.
Is ETLantic 0.53 production-supported?¶
ETLantic 0.53.0 is Beta, community-supported, with no SLA. Use it for
documented single-tenant pilots. You can embed an HTTP control plane
(etlantic-fastapi) with Supported isolation profiles
(isolated-deployment, dedicated-schema). There is no hosted multi-tenant
SaaS. See Capabilities and
Production readiness.
Available vs Experimental?¶
Available means it is in the 0.53 product envelope: documented, tested, and
covered by the Beta support policy (current minor only, no SLA).
Experimental means Alpha extras (Kafka, Iceberg, MCP, DataFusion, k8s,
Spark Connect, shared-service isolation) that may change or stay fake-first.
See Experimental surfaces.
What is the embeddable HTTP API (CP1)?¶
CP1 is the 0.39 identity and HTTP foundation: typed context, authorization,
durable accept receipts, and resumable SSE via etlantic.control_plane and
optional etlantic-fastapi (ETLanticAPI). Later minors added persistence
(CP2), durable work (CP3), policy/audit (CP4), and 0.43 CP-GA graduation
for Supported isolation profiles. The names are internal program labels, not
separate products. There is still no hosted SaaS.
See Embeddable HTTP API and
What's new in 0.43.
What does 0.53 not do?¶
Proposals never apply files, submit runs, or grant tools. There is no write MCP in core, no vendor AI SDK, and no silent optimizer. Schedules never embed payloads or secrets. Adaptive execution remains Experimental and restricted to fixture-qualified local static-batch plans; durable and remote adaptive execution are unavailable. See What's new in 0.53.
Thin FastAPI app vs control plane?¶
create_reference_app is a thin, non-CP authoring/service demo (sync
AuthoringService). The control plane is ETLanticAPI / include_router /
create_app with injected stores and authz. Do not treat the reference app
as durable multi-tenant isolation. Package README:
etlantic-fastapi.
Landing zone vs CsvStorage?¶
Landing-zone extract uses local-files bindings (root_ref, glob,
snapshot/incremental modes) for directory-shaped inputs — see
Landing zone. CsvStorage (and JSON
storage) are built-in asset bindings for named CSV/JSON files under a profile
— different job. Landing is not a replacement for CsvStorage.
Where is continuous directory watch?¶
Not in core. Continuous watching is an optional submitter outside
src/etlantic/ (for example
etlantic_fastapi.landing_sensor.LandingWatchSubmitter or
examples/landing_zone_watch_submitter.py). Snapshot/incremental landing
reads ship; long-lived watch loops do not live under core.
How does it compare to dbt / Prefect / Pandera?¶
| Tool | Primary job | Relationship |
|---|---|---|
| dbt | SQL warehouse projects | Complementary |
| Prefect / Dagster / Airflow | Orchestration | Complementary (Airflow compile; Prefect local MVP) |
| Pandera / GE | Row/table validation libraries | Complementary (ETLantic validates wiring/contracts) |
Full table and when not to use ETLantic: Compare.
Can one transformation run on multiple engines?¶
Yes with @Transformation.portable and matching plugins. Coverage differs —
see Portable Compiler Matrix.
Builders / JSON without classes?¶
Yes since 0.24. See Programmatic authoring.
Why does from etlantic import SomeError fail?¶
Many root aliases were removed in 0.26/0.27. Import from owning modules
(etlantic.exceptions, etlantic.runtime, …). See
Exceptions.
Does ETLantic include medallion bronze/silver/gold?¶
No — those stay in SparkForge / medallantic. See
Medallantic and
Migration 0.35 → 0.36.
Can I build a GUI?¶
Not as a shipped product. Use programmatic authoring plus optional
etlantic-fastapi: prefer ETLanticAPI for the embeddable HTTP API, or
create_reference_app for the thin non-CP authoring demo. See
Embeddable HTTP API,
Control plane API, and
Application integration.
Where do lineage / Graphviz diagrams come from?¶
Pipeline.to_mermaid(), etlantic viz, and Graphviz DOT / HTML exporters.
See Visualization.