API — Protocols¶
Generated from package source. Hub: Python API Reference.
Dataframe protocol¶
etlantic.dataframe
¶
Versioned dataframe execution protocol (etlantic.dataframe/1).
Core stays engine-free. Polars/Pandas plugins implement DataframePlugin.
ArtifactOwnership
¶
Bases: StrEnum
Ownership state for a dataframe artifact handle.
DataframeExecutionContext
dataclass
¶
Caller identity and plan-resolved settings for one step invocation.
DataframeMetrics
dataclass
¶
Structured metrics emitted by a dataframe step.
DataframeOutputBundle
dataclass
¶
Normalized outputs from a dataframe plugin invocation.
DataframePhase
¶
Bases: StrEnum
Reportable phases of a dataframe step execution.
DataframePlugin
¶
Bases: Protocol
Protocol for in-process dataframe execution backends.
Plugins materialize inputs, invoke registered implementations, validate frames against ContractModel types, and export records — without changing portable pipeline semantics.
materialize_input(value, *, contract_type, context, port_name)
¶
Convert a logical/runtime value into a native frame for this engine.
invoke(*, callable_, inputs, parameters, context)
¶
Invoke a registered transformation implementation.
normalize_output(result, *, output_ports, context)
¶
Normalize a callable result into valid/invalid/side outputs.
validate_frame(value, *, contract_type, context, boundary, port_name=None)
¶
Validate a frame (or records) against a contract.
Returns (value, decision, diagnostics, invalid_value).
invalid_value is set when reject/quarantine splits rows.
inspect_schema(value, *, identity)
¶
Return a NormalizedSchema-compatible mapping, if inspectable.
ensure_ownership(value, *, ownership, context)
¶
Return a handle that respects ownership / mutation isolation.
collect_if_needed(value, *, context)
¶
Collect lazy values when the plan declares a collection boundary.
to_records(value, *, contract_type)
¶
Convert a native frame to Python records for storage/publication.
row_count(value)
¶
Best-effort row count without forcing a full materialization when possible.
DataframePluginInfo
dataclass
¶
Installed dataframe plugin metadata.
DataframeValidationOutcome
¶
Bases: StrEnum
Configured outcome when invalid rows are identified.
DataframeValidationPolicy
dataclass
¶
Input/output validation policy for a dataframe step.
ValidationDecision
¶
Bases: StrEnum
Decision applied during a validation phase.
arrow_available()
¶
Return True when pyarrow can be imported.
from_arrow_table(table, *, engine)
¶
Convert an Arrow table into records (core) or raise for engine frames.
Core keeps Arrow↔records only. Engine-native construction belongs in
dataframe plugins (pl.from_arrow / Table.to_pandas). Pass
engine="records" (or any non-plugin name) to get a pylist.
records_to_arrow_table(records, *, contract_type=None)
¶
Build a pyarrow.Table from Python records.
Raises ImportError when PyArrow is not installed.
to_arrow_table(value)
¶
Best-effort conversion of a native frame to pyarrow.Table.
Returns None when the value cannot be converted without engine plugins.
Gate A interchange boundaries must use :func:to_arrow_table_strict
instead so a planned Arrow conversion cannot silently fall back.
discover_dataframe_plugins(*, profile=None)
¶
Discover dataframe plugins with authorize-before-load (0.20).
When profile is omitted, allowlists are open (non-production behavior).
Production profiles require manifests and a non-empty allowlist.
register_discovered_plugins(registry, *, plugins=None, profile=None)
¶
Register discovered dataframe plugins into a planning registry.
SQL protocol¶
etlantic.sql
¶
Versioned SQL execution protocol (etlantic.sql/1).
Core stays driver-free. Install etlantic-sql for the PostgreSQL
reference plugin.
AliasedExpr
dataclass
¶
Projection expression with output alias.
AtomicPublicationStrategy
¶
Bases: StrEnum
How replace/snapshot must publish atomically.
BinaryExpr
dataclass
¶
Binary expression over portable operators.
CallExpr
dataclass
¶
Function call expression (scalar or aggregate).
CaseWhenExpr
dataclass
¶
Searched CASE expression.
ColumnRef
dataclass
¶
Column reference within a relation alias.
CompiledSql
dataclass
¶
Compiled statement with secret-free parameter metadata.
ConcatExpr
dataclass
¶
String concatenation of columns / literals.
CteDef
dataclass
¶
Common table expression wrapping a subquery.
JoinClause
dataclass
¶
JOIN clause against another relation (fail-collision policy only).
LiteralExpr
dataclass
¶
Typed literal (serialized into bound parameters, never interpolated).
OrderByItem
dataclass
¶
ORDER BY key with null placement.
RelationRef
dataclass
¶
Logical relation identity without credentials or live connections.
parse(value)
classmethod
¶
Parse catalog.schema.name or schema.name or name.
SqlExecutionContext
dataclass
¶
Runtime identity and connection binding for SQL work.
SqlExecutionResult
dataclass
¶
Result of compiling/executing SQL work.
SqlMetrics
dataclass
¶
Normalized SQL execution metrics (no query values).
SqlParameter
dataclass
¶
Named parameter placeholder (value resolved only at runtime).
SqlPhase
¶
Bases: StrEnum
Reportable phases of SQL region / step execution.
SqlPlugin
¶
Bases: Protocol
Protocol for SQL execution backends.
Plugins compile portable :class:~etlantic.sql.protocol.SqlQuery and
:class:~etlantic.sql.protocol.SqlWrite intents to dialect SQL, execute
with parameterized bindings, and expose catalog helpers for planning.
capabilities()
¶
Return declared SQL / dialect capabilities.
quote_identifier(name)
¶
Validate and quote an identifier; raise on illegal names.
relation_from_binding(*, binding, location, metadata=None)
¶
Map a storage/binding location to a RelationRef.
compile_query(query, *, context)
¶
Compile a portable query to parameterized dialect SQL.
compile_write(write, *, context)
¶
Compile a write / publication intent.
execute(compiled, *, params, context, fetch=False)
¶
Execute compiled statements.
When fetch is False (SQL-to-SQL), must not materialize rows into
Python. When True (hybrid boundary), may return records.
execute_write(write, *, params, context)
¶
Compile and execute a write intent.
materialize_temp(query, *, temp_name, params, context)
¶
Create a temporary relation from a query without fetching rows.
load_records(records, *, target, context)
¶
Load Python/dataframe records into a staging relation.
fetch_records(relation, *, params, context, contract_type=None)
¶
Fetch rows at a hybrid boundary (counts toward rows_fetched).
inspect_relation(relation, *, context)
¶
Inspect catalog metadata without reading source rows.
rows_fetched_total()
¶
Instrumentation: cumulative rows fetched into Python.
SqlPluginInfo
dataclass
¶
Installed SQL plugin metadata.
SqlQuery
dataclass
¶
Closed portable select query IR (kernel + relational /1 surface).
SqlWrite
dataclass
¶
Write / publication intent against a target relation.
TransactionOutcome
¶
Bases: StrEnum
Known transaction outcomes for retry gating.
TrustedSqlFragment
dataclass
¶
Escape hatch for trusted SQL text (policy-gated; disabled by default).
UnaryExpr
dataclass
¶
Unary expression (NOT, negate).
WriteIntentKind
¶
Bases: StrEnum
Portable SQL write / publication intents.
discover_sql_plugins(*, profile=None)
¶
Discover SQL plugins with authorize-before-load (0.20).
load_sql_plugin(engine='sql', *, profile=None)
¶
Return a discovered plugin for engine, or None.
register_discovered_plugins(registry, *, plugins=None, profile=None)
¶
Register discovered SQL plugins into a planning registry.
alias(expr, name)
¶
Alias a projection expression.
col(name, relation=None)
¶
Column reference.
concat(*parts, separator=' ', as_=None)
¶
Concatenate string parts (columns or literals).
lit(value, *, sql_type=None)
¶
Literal that will be bound as a parameter.
select(*columns, source, where=None, limit=None, distinct=False, parameters=())
¶
Build a portable select query.
trusted_sql(text, *, param_names=(), allowed=False)
¶
Trusted SQL fragment; allowed must be True under profile policy.
is_safe_identifier(name)
¶
Return True when name is a simple unquoted-safe identifier.
redact_params(names)
¶
Build redacted parameter metadata for plans/logs.
require_safe_identifier(name)
¶
Validate identifier; raise ValueError when unsafe.
sqlmodel_table_to_relation(table)
¶
Optional SQLModel table → RelationRef + schema metadata.
Imports SQLModel only when called. Core never depends on SQLModel.
Spark protocol¶
etlantic.spark
¶
Versioned Spark execution protocol (etlantic.spark/1).
Core stays PySpark-free. Install etlantic-pyspark for the reference plugin.
Structured Streaming APIs are experimental; see :data:~etlantic.spark.STREAMING_STABILITY.
CompiledSparkPlan
dataclass
¶
Compiled Spark region — secret-free and inspectable.
DatasetRef
dataclass
¶
Logical Spark dataset identity without credentials or live sessions.
ExpressionStrategy
¶
Bases: StrEnum
How a compiled step realizes expressions.
SchemaCompatibility
¶
Bases: StrEnum
Contract ↔ Spark schema mapping outcome (never guess).
SparkAction
dataclass
¶
An explicit action boundary recorded in compiled plans.
SparkActionKind
¶
Bases: StrEnum
Explicit Spark actions (no hidden collect/show/toPandas).
SparkCompilationContext
dataclass
¶
Inputs for region compilation (no live session).
SparkDataFrameHandle
dataclass
¶
Portable handle wrapping a native Spark DataFrame identity.
The live frame lives only inside the plugin/runtime; plans and reports never embed sessions or credentials.
SparkExecutionContext
dataclass
¶
Runtime context for Spark execution.
SparkExecutionResult
dataclass
¶
Result of executing a compiled Spark plan or step.
SparkMetrics
dataclass
¶
Normalized Spark execution metrics (no secret values).
SparkPhase
¶
Bases: StrEnum
Reportable phases of Spark region / step execution.
SparkPlanRegion
dataclass
¶
Planner-facing description of a Spark execution region.
SparkPlugin
¶
Bases: Protocol
PySpark / Spark execution plugin protocol.
Implementations compile portable Spark plan regions, execute transformations
and writes against a live SparkSession, and expose schema inspection and
record export for hybrid boundaries.
info
property
¶
Plugin metadata including engine id and declared capabilities.
capabilities()
¶
Return Spark-specific capability flags for planning.
dataset_from_binding(*, binding, location=None, metadata=None)
¶
Resolve a profile asset/binding to a logical dataset reference.
compile(region, *, context)
¶
Lower a portable Spark plan region to engine-specific actions.
execute(compiled, *, context, inputs=None)
¶
Execute a compiled plan region and return metrics plus outputs.
execute_step(*, callable_, inputs, params, context)
¶
Invoke a native Spark transformation implementation.
execute_write(write, *, context)
¶
Execute a write/publication intent against Spark storage.
inspect_schema(value, *, contract_type=None)
¶
Return a NormalizedSchema-compatible mapping for a DataFrame handle.
to_records(value, *, contract_type=None)
¶
Materialize rows as Python records for storage or publication.
split_valid_invalid(value, *, contract_type, context)
¶
Split a DataFrame into valid and invalid partitions by contract.
SparkPluginInfo
dataclass
¶
Discoverable plugin metadata.
SparkUdfPolicy
¶
Bases: StrEnum
Profile policy for UDF usage during planning/compile.
SparkWrite
dataclass
¶
Portable write intent realized by the Spark/Delta plugin.
SparkWriteMode
¶
Bases: StrEnum
Delta-compatible portable write modes.
ResourceContext
dataclass
¶
Runtime context for acquire/release (may carry secret resolvers).
SessionOwnership
¶
Bases: StrEnum
Who owns the SparkSession lifecycle.
SparkProvider
¶
Bases: Protocol
Supplies SparkSession lifecycle without embedding secrets in plans.
SparkProviderInfo
dataclass
¶
Discoverable provider metadata.
SparkSessionHandle
dataclass
¶
Opaque handle for an acquired session (no credentials).
SparkSessionRequest
dataclass
¶
Secret-free request for a Spark session.
Credentials and secret values are resolved by the provider at acquire time
via secret_refs / runtime secret providers — never embedded here.
FieldMapping
dataclass
¶
One field mapping outcome.
SchemaMappingResult
dataclass
¶
Aggregate contract ↔ Spark schema mapping.
LateEventPolicy
¶
Bases: StrEnum
Policy for events past the watermark.
StreamingOutputMode
¶
Bases: StrEnum
Streaming output modes.
StreamingProgress
dataclass
¶
Normalized streaming progress evidence.
StreamingQuerySpec
dataclass
¶
Experimental streaming query declaration.
StreamingTrigger
¶
Bases: StrEnum
Supported Structured Streaming triggers.
WatermarkSpec
dataclass
¶
Event-time watermark configuration.
discover_spark_plugins(*, profile=None)
¶
Discover Spark plugins with authorize-before-load (0.20).
discover_spark_providers(*, profile=None)
¶
Discover Spark providers with authorize-before-load (0.20).
load_spark_plugin(engine='pyspark', *, profile=None)
¶
Return a discovered plugin for engine, or None.
load_spark_provider(name='local', *, profile=None)
¶
Return a discovered provider by name, or None.
register_discovered_plugins(registry, *, plugins=None, profile=None)
¶
Register discovered Spark plugins into a planning registry.
compare_types(logical, spark_type)
¶
Compare logical vs observed Spark type without guessing.
map_contract_schema(contract_type, *, observed=None)
¶
Map a ContractModel / Pydantic model to Spark types with explicit outcomes.
observation_from_spark_schema(spark_schema, *, source='spark', partition_columns=None)
¶
Build a normalized schema observation from a Spark StructType-like object.
Orchestration protocol¶
etlantic.orchestration
¶
External orchestration protocols (etlantic.orchestration/1).
Compile secret-free :class:~etlantic.plan.model.PipelinePlan objects into
orchestrator artifacts via :func:~etlantic.orchestration.compile.compile_plan.
Install etlantic-airflow for the reference Airflow compiler.
ArtifactTransportPolicy
dataclass
¶
Policy controlling how step outputs cross orchestrator task boundaries.
OrchestrationCompilationError
¶
CancellationResult
dataclass
¶
Result of requesting cancellation of an external run.
CorrelationKeys
dataclass
¶
Keys linking ETLantic runs to backend orchestrator identities.
PollResult
dataclass
¶
Polled status for an external orchestration run.
SubmissionResult
dataclass
¶
Result of submitting a compiled artifact to an orchestrator.
SubmissionStatus
¶
Bases: StrEnum
Lifecycle of an externally submitted orchestration run.
CompilationContext
dataclass
¶
Context supplied to orchestrator compilers.
CompilationDiagnostic
dataclass
¶
Structured diagnostic from orchestration compilation.
CompiledOrchestrationArtifact
dataclass
¶
Deterministic compiled orchestration artifact (secret-free).
write(path)
¶
Write the generated module source to path.
CompiledTask
dataclass
¶
Backend-agnostic compiled task derived from a plan node.
ExecutionIntent
dataclass
¶
Portable execution settings for orchestration backends.
OrchestrationPhase
¶
Bases: StrEnum
Phases of external orchestration.
OrchestratorPlugin
¶
Bases: Protocol
Protocol for external orchestration compilers.
Implementations turn a resolved :class:~etlantic.plan.model.PipelinePlan
into backend artifacts (for example Airflow DAG modules) without executing
user transformation code during compilation.
info
property
¶
Plugin metadata including orchestrator id and capabilities.
capabilities()
¶
Return scheduling/retry/parallelism flags for planning.
compile(plan, *, context)
¶
Compile a pipeline plan into a deterministic orchestration artifact.
explain(artifact)
¶
Return a JSON-serializable summary of a compiled artifact.
OrchestratorPluginInfo
dataclass
¶
Identity and capability declaration for an orchestrator plugin.
ScheduleIntent
dataclass
¶
Portable scheduling intent (orchestrator maps to backend constructs).
TaskRetryPolicy
¶
Bases: StrEnum
How retries are applied on a compiled task.
validate_artifact_for_transport(artifact, *, estimated_bytes=None, policy=None)
¶
Return diagnostics when an artifact cannot safely cross task boundaries.
xcom_safe_payload(artifact)
¶
Return a small, secret-free payload suitable for XCom / task messaging.
compile_plan(plan, *, target='airflow', profile=None, plugin=None, context=None, plugins=None, allow_adhoc_profile=False)
¶
Compile a secret-free :class:~etlantic.plan.model.PipelinePlan into an orchestration artifact.
Verifies the plan fingerprint, resolves the target orchestrator plugin
(discovered entry points or explicit plugin), and fails closed when
required capabilities are missing or compilation diagnostics contain errors.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plan
|
PipelinePlan
|
Resolved plan to compile. Must include a valid |
required |
target
|
str
|
Orchestrator engine id (default |
'airflow'
|
profile
|
Profile | str | None
|
Optional profile for compilation context and plugin discovery. |
None
|
plugin
|
OrchestratorPlugin | None
|
Optional pre-constructed orchestrator plugin instance. |
None
|
context
|
CompilationContext | None
|
Optional explicit :class: |
None
|
plugins
|
dict[str, OrchestratorPlugin] | None
|
Optional pre-discovered plugin mapping ( |
None
|
allow_adhoc_profile
|
bool
|
Forwarded to :func: |
False
|
Returns:
| Type | Description |
|---|---|
CompiledOrchestrationArtifact
|
class: |
CompiledOrchestrationArtifact
|
with secret-free source/metadata suitable for DAG check-in. |
Raises:
| Type | Description |
|---|---|
ValueError
|
When :func: |
OrchestrationCompilationError
|
When the plugin is missing or compilation
diagnostics include errors (codes such as |
explain_compilation(artifact)
¶
Return a JSON-friendly compilation explanation.
discover_orchestrator_plugins(*, profile=None)
¶
Discover orchestrator plugins with authorize-before-load (0.20).
load_orchestrator_plugin(engine='airflow', *, profile=None)
¶
Return a discovered orchestrator plugin for engine, or None.
plugin_registry_snapshot(*, profile=None)
¶
Return serializable descriptors for discovered orchestrator plugins.
register_discovered_plugins(registry, *, plugins=None, profile=None)
¶
Register discovered orchestrator plugins into a planning registry.
comparable_report_shape(report)
¶
Normalized shape for comparing local vs orchestrated runs.
correlate_poll_to_report(report, poll)
¶
Attach backend correlation and task states onto a normalized report.
context_from_profile(profile, *, target, max_inline_bytes=65536)
¶
Build a compilation context from a profile.
dag_id_for_plan(plan)
¶
Deterministic DAG / workflow id from plan identity.
execution_from_profile(profile)
¶
Extract portable execution intent from profile fields/metadata.
map_plan_to_tasks(plan, *, context)
¶
Map logical graph nodes/edges to a compiled task graph.
schedule_from_profile(profile)
¶
Extract portable schedule intent from profile fields/metadata.
apply_reliability_to_task(task, *, plan, execution)
¶
Return an updated task with retry + reliability fields applied.
reliability_metadata(plan, node_name)
¶
Collect portable repair/backfill/reconciliation fields for a node.
resolve_task_retry(*, node_name, node_kind, plan, execution, declarations=None)
¶
Map profile retry intent to a safe per-task policy.
Unsafe sinks with requested retries produce PMORCH310 and force retries
off (or reject when strict metadata asks).
Visualization¶
etlantic.viz
¶
Shared graph IR and visualization exporters beyond Mermaid.
Public helpers:
- :func:
logical_graph_to_ir/ :func:plan_to_ir— build a :class:GraphIR - :func:
graph_to_dot— Graphviz DOT text - :func:
graph_to_html— single-page HTML lineage document - :func:
lineage_export— JSON lineage document (etlantic.lineage/1)
CLI: etlantic viz dot|html|lineage.
GraphNode
dataclass
¶
Node in a visualization intermediate representation (IR).
GraphEdge
dataclass
¶
Directed edge in a visualization IR.
GraphIR
dataclass
¶
Backend-neutral graph intermediate representation.
logical_graph_to_ir(graph)
¶
Convert a logical pipeline graph into visualization IR.
plan_to_ir(plan)
¶
Convert a resolved plan's logical graph into visualization IR.
graph_to_dot(ir)
¶
Export Graphviz DOT (no graphviz binary required to generate text).
ir_to_logical_graph(ir)
¶
Rebuild a LogicalGraph suitable for Mermaid rendering.
graph_to_html(ir, *, include_mermaid=True)
¶
Simple HTML lineage/docs page (stdlib only).
lineage_export(ir)
¶
JSON lineage document suitable for docs / agents.
etlantic.mermaid
¶
Mermaid diagram generation from logical graphs.
graph_to_mermaid(graph)
¶
Render a logical pipeline graph as a Mermaid flowchart.
Output is deterministic for a given graph.
Agents, IDE, and notebooks¶
etlantic.agents
¶
Agent guidance generators (AGENTS.md, CLAUDE.md, Codex SKILL, Cursor rules).
generate_agent_guidance(root, *, overwrite=True)
¶
Write agent guidance files under root.
etlantic.ide
¶
etlantic.notebook
¶
Observability¶
etlantic.observability
¶
Schema history¶
etlantic.schema_history
¶
File-backed schema history provider (no source rows).
FileSchemaHistoryProvider
dataclass
¶
Canonical-file schema history under a root directory.
Observations are fingerprints and field metadata only — never source rows.
Writes go through :class:SafeIoPolicy (0.20).
acknowledge(subject_id, *, note=None)
¶
Record an acknowledgment without mutating the contract.
assert_no_row_payload(observation)
¶
Refuse observations that embed source-row-like payloads.
Capabilities¶
etlantic.capabilities
¶
Plugin capability declarations and negotiation results.
CapabilityDecision
¶
Bases: StrEnum
Outcome of comparing required vs available capabilities.
PluginCapabilities
dataclass
¶
Declared capabilities of a plugin or engine.
Dataframe-oriented flags (eager/lazy/arrow/...) are first-class for 0.5
planning. SQL-oriented flags are first-class for 0.6. Unknown requirements
may still be declared via extras.
CapabilityNegotiation
dataclass
¶
Record of a capability check for one requirement.
to_dict()
¶
Serialize negotiation record.
negotiate_capabilities(*, requirements, available, fallback=None, allow_fallback=False)
¶
Negotiate required capabilities against an available engine.
Unsupported requirements fail closed unless allow_fallback is True and
a fallback engine covers the requirement.
Reliability and schema drift¶
etlantic.reliability
¶
Portable reliability and intent models (schemas in 0.3; enforcement in 0.4+).
WriteMode
¶
Bases: StrEnum
Declared write semantics.
MaterializationMode
¶
Bases: StrEnum
Declared materialization intent.
FreshnessExpectation
dataclass
¶
PartitionCompletenessExpectation
dataclass
¶
WriteIntent
dataclass
¶
MaterializationIntent
dataclass
¶
IdempotencyDeclaration
dataclass
¶
RetrySafetyDeclaration
dataclass
¶
ReconciliationDeclaration
dataclass
¶
BackfillDeclaration
dataclass
¶
RepairDeclaration
dataclass
¶
ReliabilityEvidence
dataclass
¶
environment_identity(*, profile, security_domain, workspace='')
¶
Deterministic identity for a resolved environment.
implementation_selection_identity(*, transformation_id, engine)
¶
Deterministic identity for a selected implementation.
quality_metric_identity(*, subject_id, metric)
¶
Deterministic identity for a quality metric.
statistical_observation_identity(*, subject_id, feature)
¶
Deterministic identity for a statistical observation.
fingerprint_mapping(data)
¶
Deterministic fingerprint for a mapping.
etlantic.reliability_runtime
¶
Runtime reliability enforcement helpers.
BackfillRequest
dataclass
¶
Runtime backfill request derived from intent/declaration.
coerce_observed_at(value)
¶
Parse an observed timestamp from metadata or provider payloads.
resolve_freshness_observed_at(expectation, *, node_name, binding, metadata)
¶
Resolve observed_at for a freshness check from run metadata.
Lookup order:
1. metadata["freshness_observed_at"][node_name|binding]
2. expectation.metadata["observed_at"]
check_freshness(expectation, *, observed_at, now=None)
¶
Local freshness check against a declared expectation.
minimum_safe_repair(*, failed_nodes, downstream)
¶
Compute minimum-safe repair closure from failed nodes.
etlantic.schema_drift
¶
Schema drift models: normalized schema, observations, changes, impact (0.3).
DriftImpact
¶
Bases: StrEnum
Impact vocabulary for schema changes.
NormalizedSchema
dataclass
¶
SchemaObservation
dataclass
¶
Immutable schema observation (no row data or secrets).
to_dict()
¶
Serialize observation.
normalize_schema_from_model(model, *, identity=None)
¶
Normalize a ContractModel/Data class into a logical schema.
normalize_schema_from_fields(fields, *, identity)
¶
Normalize a list of field mappings (operational observation path).
diff_normalized_schemas(baseline, candidate)
¶
Operational drift path: compare normalized schemas without ContractModel.
diff_contract_schemas(previous, current)
¶
Contract-drift path: delegate compatibility meaning to ContractModel diffs.
etlantic.schema_policy
¶
Schema drift policy decisions for local runtime.
Testing helpers¶
etlantic.testing
¶
ETLantic testing helpers.
assert_plugin_info(plugin, *, engine)
¶
Assert a dataframe plugin advertises the expected engine and protocol.
assert_roundtrip_records(plugin, *, rows, contract_type=None)
¶
Assert materialize → validate → to_records preserves row dicts.
run_conformance_suite(plugin, *, engine, sample_rows, contract_type=None)
¶
Minimal conformance checks for third-party dataframe plugins.
run_tabular_interchange_conformance_smoke(producer_caps, consumer_caps)
¶
Select and round-trip a descriptor from two capability declarations.
assert_orchestrator_plugin_info(plugin, *, engine)
¶
Assert an orchestrator plugin advertises the expected engine and protocol.
run_orchestrator_conformance_suite(plugin, *, engine, plan, context=None)
¶
Compile a plan and assert a secret-free artifact is produced.
normalize_rows(rows)
¶
Stable cross-engine row normalization for conformance compares.
run_portable_transform_conformance_suite(compiler, *, profiles=None, to_frame=None, enforce_fixture_coverage=True)
¶
Run capability-selected portable transform conformance for compiler.
The suite selects mandatory fixtures for every advertised profile/action/
function claim. Compilers must pass every selected fixture (or correctly
reject unsupported modes at analyze time). When
enforce_fixture_coverage is true, claiming kernel/relational profiles
or individual actions/functions without a matching fixture fails the suite.
assert_scheduler_plugin_info(scheduler)
¶
Assert a scheduler plugin advertises direct-execution protocol metadata.
run_scheduler_conformance_suite(scheduler)
¶
Minimal public suite shared by LocalScheduler and plugin schedulers.
assert_missing_secret_fails(provider, reference)
¶
Providers must fail closed when a secret is absent.
assert_secret_provider_info(provider)
¶
Assert a secret provider exposes a non-empty descriptor and capabilities.
run_secret_conformance_suite(provider, *, reference, expected_substring=None)
¶
Resolve one secret and assert SecretValue hygiene.
assert_sql_plugin_info(plugin)
¶
Assert a SQL plugin advertises protocol version and core capabilities.
run_sql_conformance_suite(plugin)
¶
Minimal conformance checks for SQL plugins (driver-backed).
assert_write_intent_parity(backends, *, subject_id, mode)
¶
Assert each backend can declare the same write intent vocabulary.
backends maps engine name → object with supports_write_mode(mode)
or a capabilities mapping containing write mode strings.
run_write_semantics_parity_suite(backends, *, subject_id='parity', modes=(WriteMode.APPEND, WriteMode.OVERWRITE))
¶
Run parity checks for shared write intents across backends.
Exceptions¶
etlantic.exceptions
¶
Public exception hierarchy for ETLantic.
ETLanticError
¶
Bases: Exception
Base class for public ETLantic exceptions.
ModelDefinitionError
¶
Bases: ETLanticError
Raised when a class definition cannot form a usable model.
PipelineValidationError
¶
InternalETLanticError
¶
Bases: ETLanticError
Raised when an internal invariant is violated.