Python API Reference¶
Status: Available in ETLantic 0.15.0. Signatures and docstrings below are generated from the package source.
Start here by persona¶
| Persona | Start with | Then |
|---|---|---|
| Pipeline author | Root imports below, CLI | etlantic.pipeline, etlantic.plan, etlantic.reports |
| Plugin author | etlantic.dataframe / .sql / .spark / .orchestration / .secrets |
Testing, Plugin SDK |
| CI / ops | CLI, Runtime configuration | etlantic.plugin_trust, SARIF validate |
The package root is the supported convenience import surface for common authoring, planning, runtime, storage, report, secret, and interchange types:
from etlantic import (
Data,
Input,
Output,
Parameter,
Pipeline,
PipelineRuntime,
Load,
Extract,
Transformation,
)
DataContractModel is a deprecated alias for Data.
Authoring¶
Portable authoring and compilers (0.11–0.14)
etlantic.transform, @Transformation.portable, symbolic DataFrame and
Column objects, and functions as F normalize to published DTCS 3.0
dtcs.transform-plan/2 models. etlantic.transform.compiler defines the
etlantic.transform-compiler/1 protocol. Official compilers ship in
etlantic-polars, etlantic-pyspark, and etlantic-pandas (eager). See
package READMEs under packages/ for optional-package APIs (MkDocs scans
src/ only). Also see
Portable Transformations
and Portable Transform Compiler.
Core behavioral contracts¶
The generated signatures below are supplemented by these current guarantees:
| API | Returns | Important failures / side effects |
|---|---|---|
Transformation.step(**bindings) |
A symbolic Step; no user code runs |
Unknown bindings raise ModelDefinitionError; required ports are validated before execution |
Transformation.implementation(engine) |
A decorator returning the original callable | Registration replaces the implementation for the same class/engine in the current process |
Transformation.portable |
Decorator registering a symbolic definition | Authoring errors raise ModelDefinitionError (PMXFORM*) at registration; does not execute |
Transformation.to_transform_plan() |
Deep-copied dtcs.transform-plan/2 dict |
Raises ModelDefinitionError if no portable definition is registered |
Transformation.portable_fingerprint() |
Hex fingerprint string | Same failure mode as to_transform_plan |
Pipeline.validate(...) |
ValidationReport |
Does not execute transformation implementations; production empty allowlist fails closed (PMPLUG401) |
Pipeline.plan(...) |
Immutable, secret-free PipelinePlan |
Missing plugins, bindings, trust, or capabilities produce planning/validation failure |
Pipeline.run(...) |
PipelineRunReport |
Executes in-process; storage and plugin side effects follow the resolved plan |
Pipeline.arun(...) |
Awaitable PipelineRunReport |
Uses the same validation and planning path as run |
Pipeline.to_mermaid() |
Mermaid flowchart string | Builds the logical graph but does not plan or execute |
Minimal validation pattern:
report = CustomerPipeline.validate(profile="development")
report.raise_for_errors()
plan = CustomerPipeline.plan(profile="development")
Minimal execution pattern:
runtime = PipelineRuntime()
runtime.memory.seed("customer_source", records)
run_report = CustomerPipeline.run(
profile="development",
runtime=runtime,
)
if run_report.status.value != "succeeded":
raise RuntimeError(run_report.to_text())
PipelineRuntime is application-owned. A new Python or CLI process receives a
new process-local memory store and report store unless durable providers are
configured.
Data contracts¶
etlantic.contracts
¶
Data-contract integration boundary for ContractModel.
Data is ETLantic's thin public facade over ContractModel (DD-010A).
ContractModel retains authority for data-contract semantics and ODCS.
DataContractModel remains as a deprecated compatibility alias.
is_data_contract_type(obj)
¶
Return True when obj is a ContractModel-compatible data-contract class.
resolve_contract_type(annotation)
¶
Extract a data-contract class from a type annotation when possible.
Returns None when the annotation is not a concrete ContractModel subclass.
load_data_contract(path, *, root=None, class_name=None)
¶
Load an ODCS artifact into a Data (ContractModel) subclass.
write_odcs(model, path, *, root=None)
¶
Write a Data class to an ODCS YAML file.
Transformations¶
etlantic.transformation
¶
Transformation contracts, steps, and implementation registration.
PortDefinition
dataclass
¶
Introspected port on a transformation class.
ImplementationRecord
dataclass
¶
Registered execution implementation for a transformation.
Step
dataclass
¶
A concrete use of a transformation inside a pipeline.
Created via Transformation.step(...). Attribute access to output port
names yields :class:OutputRef values for downstream wiring.
Transformation
¶
Base class for typed transformation contracts.
Subclasses declare Input, Output, and Parameter annotations.
Implementations are registered separately with :meth:implementation.
identity()
classmethod
¶
Stable transformation identity.
inputs()
classmethod
¶
Return declared input ports in definition order.
outputs()
classmethod
¶
Return declared output ports in definition order.
parameters()
classmethod
¶
Return declared parameters in definition order.
implementations()
classmethod
¶
Return registered implementations keyed by engine name.
validate_definition()
classmethod
¶
Return definition problems for this transformation class.
implementation(engine)
classmethod
¶
Register a native callable for one execution engine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
engine
|
str
|
Registry engine name such as |
required |
Returns:
| Type | Description |
|---|---|
Callable[[F], F]
|
A decorator that records the callable and returns it unchanged. |
Note
Registration is process-local. Registering the same engine again replaces the earlier callable for this transformation class.
portable(fn)
classmethod
¶
Register a portable symbolic definition for this transformation.
The callable is invoked with symbolic FrameExpr inputs and
ParameterRef parameters during trusted import. It must return a
FrameExpr or a mapping of declared output names to FrameExpr.
Native :meth:implementation callables remain available as escape
hatches. Portable definitions emit dtcs.transform-plan/2 and do not
execute data.
portable_definition()
classmethod
¶
Return the cached :class:~etlantic.transform.PortableDefinition, if any.
to_transform_plan()
classmethod
¶
Return the portable dtcs.transform-plan/2 document.
Raises:
| Type | Description |
|---|---|
ModelDefinitionError
|
If no portable definition is registered. |
portable_fingerprint()
classmethod
¶
Return the portable plan fingerprint.
to_dtcs()
classmethod
¶
Generate a DTCS document dict for this transformation.
from_dtcs(source, *, contracts=None, root=None, class_name=None)
classmethod
¶
Load a Transformation subclass from a DTCS artifact.
step(**kwargs)
classmethod
¶
Create a symbolic use of this transformation inside a pipeline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
**kwargs
|
Any
|
Input bindings and parameter values named exactly as the subclass declarations. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
A |
Step
|
class: |
Raises:
| Type | Description |
|---|---|
ModelDefinitionError
|
If a keyword does not name a declared input or parameter, or required declarations cannot be satisfied. |
This method does not execute a registered implementation.
Portable transform authoring (etlantic.transform)¶
etlantic.transform
¶
Portable PySpark-inspired transformation authoring (etlantic.transform/1).
ColumnExpr
dataclass
¶
Symbolic column or scalar expression that lowers to a DTCS structured node.
ParameterRef
dataclass
¶
Symbolic reference to a transformation Parameter port.
CompiledTransform
dataclass
¶
In-memory compiled artifact (never serialize backend objects into plans).
PortableTransformCompiler
¶
Bases: Protocol
Plugin protocol for analyzing, compiling, and executing portable IR.
TransformCapabilities
dataclass
¶
Advertised compiler capabilities over DTCS profiles and operators.
TransformCompileContext
dataclass
¶
Caller identity for compile() (no data access).
TransformCompilerInfo
dataclass
¶
Installed transform compiler metadata.
TransformExecutionContext
dataclass
¶
Runtime identity for execute().
TransformOutputBundle
dataclass
¶
Normalized compiler execution outputs.
TransformPlanningContext
dataclass
¶
Caller identity for analyze() (no data access).
TransformSupportFinding
dataclass
¶
One unsupported or conditional requirement from analyze().
TransformSupportReport
dataclass
¶
Deterministic support analysis for a transformation plan.
FrameExpr
dataclass
¶
Symbolic relation built from an input port or prior actions.
GroupedData
dataclass
¶
Result of groupBy awaiting aggregation.
PortableDefinition
dataclass
¶
Built portable transformation plan and authoring metadata.
TransformBudgets
dataclass
¶
Security and resource budgets for portable definition building.
Window
¶
Factory for window specifications.
WindowSpec
dataclass
¶
Immutable window specification.
input_frame(name, *, schema_fields=None)
¶
Create a FrameExpr bound to a transformation input port.
build_portable_definition(*, transformation, produced, budgets=DEFAULT_BUDGETS)
¶
Validate outputs, export dtcs.transform-plan/2, and fingerprint.
invoke_portable(transformation, fn, *, budgets=DEFAULT_BUDGETS)
¶
Call a portable authoring function with symbolic bindings and build IR.
lambda_(*parameters, body=None)
¶
Build a bounded DTCS lambda Expression node.
Prefer lambda_("element", body=lambda element: element > 0) (callable
form binds parameters with scope: "lambda"). A pre-built ColumnExpr
body is accepted only when it already uses lambda-scoped fieldRefs.
transform(collection, fn)
¶
Higher-order transform over a collection using a lambda ColumnExpr.
Symbolic only: FrameExpr / ColumnExpr trees lower to DTCS plans. They are
not Polars/Pandas/Spark objects. Polars and PySpark relational compilation
shipped in 0.13; eager Pandas relational compilation shipped in 0.14.
Portable transform compiler protocol (etlantic.transform.compiler)¶
etlantic.transform.compiler
¶
Portable transform compiler protocol (etlantic.transform-compiler/1).
TransformCapabilities
dataclass
¶
Advertised compiler capabilities over DTCS profiles and operators.
TransformCompilerInfo
dataclass
¶
Installed transform compiler metadata.
TransformSupportFinding
dataclass
¶
One unsupported or conditional requirement from analyze().
TransformSupportReport
dataclass
¶
Deterministic support analysis for a transformation plan.
TransformPlanningContext
dataclass
¶
Caller identity for analyze() (no data access).
TransformCompileContext
dataclass
¶
Caller identity for compile() (no data access).
TransformExecutionContext
dataclass
¶
Runtime identity for execute().
CompiledTransform
dataclass
¶
In-memory compiled artifact (never serialize backend objects into plans).
TransformOutputBundle
dataclass
¶
Normalized compiler execution outputs.
PortableTransformCompiler
¶
Bases: Protocol
Plugin protocol for analyzing, compiling, and executing portable IR.
Discovery helpers:
etlantic.transform.discovery
¶
Entry-point discovery for portable transform compilers.
discover_transform_compilers()
¶
Load compilers registered under etlantic.transform_compilers.
Returns engine name → compiler instance. Broken entry points are skipped with a warning (planning fails closed when the selected engine lacks a compiler and policy requires portable compilation).
register_discovered_compilers(registry, *, compilers=None)
¶
Register discovered transform compilers into a planning registry.
load_transform_compiler(engine)
¶
Return a discovered compiler for engine, or None.
compiler_registry_snapshot()
¶
Return serializable descriptors for discovered compilers.
discover_transform_compilers_for_profile(profile)
¶
Discover compilers and apply profile.plugin_allowlist trust rules.
Optional package factories (install etlantic-polars):
etlantic_polars.create_plugin, etlantic_polars.create_transform_compiler,
etlantic_polars.PolarsTransformCompiler.
Pipelines¶
etlantic.pipeline
¶
Pipeline authoring: Extract, Load, Pipeline, and subpipelines.
Extract
¶
A typed logical entry boundary that introduces data into a pipeline.
Constructing an Extract never reads data. Profiles resolve the logical
asset name to an environment-specific provider at plan/runtime time.
binding
property
¶
Deprecated alias for :attr:asset (removed in 0.16).
result
property
¶
Default output reference for this extract.
as_output_ref(*, default_port='result')
¶
Return an OutputRef for this extract's produced dataset.
bind(name, *, pipeline_id=None)
¶
Return an extract bound to a node name within a pipeline.
Load
¶
A typed logical publication boundary that receives data from a pipeline.
Constructing a Load never writes data. Profiles resolve the logical
asset name to an environment-specific provider at plan/runtime time.
SubpipelineInstance
dataclass
¶
An embedded child pipeline with parent-side bindings.
bind_name(name, *, pipeline_id=None)
¶
Bind this subpipeline instance to a parent node name.
Pipeline
¶
Declarative typed pipeline graph.
Subclasses declare Extract, transformation Step, Load, and
optional subpipeline members. Importing a pipeline does not execute it.
identity()
classmethod
¶
Stable pipeline identity.
build_graph()
classmethod
¶
Build (and cache) the immutable logical graph for this pipeline.
inspect()
classmethod
¶
Return the read-only logical graph for this pipeline.
validate(*, profile=None, policy=None, context=None)
classmethod
¶
Validate the complete graph without executing transformation code.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
str | Any
|
Built-in profile name or explicit |
None
|
policy
|
str | Any
|
Validation policy name or object. |
None
|
context
|
Any
|
Optional planning context with registries and capabilities. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A |
plan(profile=None, *, context=None, selection=None)
classmethod
¶
Resolve an immutable, secret-free execution plan.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
str | Any
|
Built-in profile name or explicit |
None
|
context
|
Any
|
Optional planning context with bindings and plugins. |
None
|
selection
|
dict[str, Any] | None
|
Optional partial-run selection mapping. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A deterministic |
Any
|
transformation code or resolve secret values. |
explain_plan(profile=None, *, context=None, selection=None)
classmethod
¶
Return a structured explanation of the planned pipeline.
run(profile='development', *, request=None, runtime=None, context=None, workspace=None)
classmethod
¶
Validate, plan, and execute this pipeline in the current process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
profile
|
str | Any
|
Built-in profile name or explicit |
'development'
|
request
|
Any
|
Optional run selection, intent, and policy request. |
None
|
runtime
|
Any
|
Application-owned |
None
|
context
|
Any
|
Optional planning context. |
None
|
workspace
|
str | Any
|
Optional durable workspace root. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
A structured |
Raises:
| Type | Description |
|---|---|
PipelineValidationError
|
If validation fails before execution. |
PipelineExecutionError
|
If execution cannot produce a run report. |
Storage writes and plugin calls follow the resolved plan. Process-local memory and report stores do not survive a new CLI or Python process.
arun(profile='development', *, request=None, runtime=None, context=None, workspace=None)
async
classmethod
¶
Validate, plan, and execute this pipeline locally (async).
debug(profile='development', *, runtime=None, context=None)
classmethod
¶
Open a stateful local debug session.
to_mermaid()
classmethod
¶
Generate a Mermaid flowchart from the logical graph.
to_dpcs()
classmethod
¶
Generate a DPCS document dict for this pipeline.
from_dpcs(source, *, registry=None, root=None, class_name=None)
classmethod
¶
Load a Pipeline subclass from a DPCS artifact.
generate_contracts()
classmethod
¶
Discover and build an in-memory contract bundle.
write_contracts(directory)
classmethod
¶
Generate and write ODCS/DTCS/DPCS artifacts under directory.
subpipeline(**bindings)
classmethod
¶
Embed this pipeline as a reusable subpipeline in a parent.
Ports and references¶
etlantic.ports
¶
Port annotation markers: Input, Output, and Parameter.
Input
¶
Bases: Generic[T]
Declares data consumed under contract T.
Output
¶
Parameter
¶
Bases: Generic[T]
Declares typed configuration that is not a graph edge.
with_default(value)
¶
Return a copy of this parameter marker with a default value.
unwrap_port_marker(annotation)
¶
Classify an annotation as Input/Output/Parameter and return (kind, contract).
Returns (None, None) when the annotation is not a port marker.
The first element is the marker class (Input, Output, or Parameter).
etlantic.refs
¶
Typed output references for pipeline wiring.
OutputRef
dataclass
¶
Bases: Generic[T]
A typed reference to a concrete producer port in a pipeline graph.
Records the producing node, named output port, and output contract. Does not hold runtime dataframe or storage objects during authoring.
as_output_ref(value, *, default_port='result')
¶
Normalize a Source, Step attribute, or OutputRef into an OutputRef.
Validation and diagnostics¶
etlantic.diagnostics
¶
Structured diagnostics and validation reports.
Severity
¶
Bases: StrEnum
Diagnostic severity levels.
SourceLocation
dataclass
¶
Optional origin of a diagnostic (file, Python object, or contract path).
DiagnosticAction
dataclass
¶
Machine-readable action a future IDE can expose as a quick fix.
to_dict()
¶
Serialize action for tooling.
Diagnostic
dataclass
¶
A structured finding from loading, inspection, or validation.
ValidationReport
dataclass
¶
Immutable collection of diagnostics for a validation pass.
valid
property
¶
Return True when no error-severity diagnostics are present.
has_errors
property
¶
Return True when at least one error-severity diagnostic is present.
errors
property
¶
Return error-severity diagnostics.
warnings
property
¶
Return warning-severity diagnostics.
filter(*, severity=None, code=None, phase=None)
¶
Return a report containing only matching diagnostics.
raise_for_errors()
¶
Raise :class:PipelineValidationError if the report is invalid.
merge(other)
¶
Combine two reports, preserving order and uniqueness by identity.
from_diagnostics(diagnostics, *, phases=())
classmethod
¶
Build a report from an iterable of diagnostics.
codes()
¶
Return diagnostic codes in report order.
etlantic.validation
¶
etlantic.policy
¶
Named validation and quality-gate policies.
PolicyMode
¶
Bases: StrEnum
How a policy treats findings.
ValidationPolicy
dataclass
¶
Named validation / quality-gate policy selected by profile or caller.
to_dict()
¶
Serialize policy.
register_validation_policy(policy)
¶
Register a named validation policy for later resolve_validation_policy.
resolve_validation_policy(name)
¶
Resolve a policy name or object.
Unknown names fail closed rather than inventing an empty default policy.
Profiles, planning, and registries¶
etlantic.profile
¶
Execution profiles that bind logical pipelines to environments.
Profile
dataclass
¶
Environment binding for planning without changing logical semantics.
assets
property
¶
Preferred public view of logical asset → provider resolution.
identity()
¶
Stable profile identity.
to_dict()
¶
Serialize profile for public JSON (assets preferred, bindings mirrored).
to_plan_snapshot()
¶
Fingerprint-stable snapshot retaining the pre-0.15 bindings shape.
from_dict(data, *, warn_legacy=True)
classmethod
¶
Deserialize a profile mapping.
with_updates(**kwargs)
¶
Return a copy with selected fields replaced.
Unknown keys raise TypeError so typos like plugin_allow_list
cannot silently leave production allowlists empty. assets is accepted
as the preferred alias for bindings.
development_profile(**overrides)
¶
Built-in development profile template.
test_profile(**overrides)
¶
Built-in test profile template.
production_profile(**overrides)
¶
Built-in production profile template.
resolve_profile(profile)
¶
Resolve a profile name, JSON path, or object to a concrete Profile.
When profile is a string ending in .json that exists as a file,
the profile is loaded with :func:load_profile. Built-in template names
(development, production, …) resolve to templates. Other bare
names create an empty :class:Profile with that name.
write_profile(profile, path)
¶
Write a profile as JSON.
load_profile(path)
¶
Load a profile from a JSON file.
etlantic.plan
¶
PipelinePlan IR package.
ArtifactRef
dataclass
¶
ArtifactStrategy
¶
Bases: StrEnum
How a logical OutputRef is realized for consumers.
PipelinePlan
dataclass
¶
Immutable, versioned, secret-free execution-facing IR.
explain_plan(plan)
¶
Return a structured, tooling-friendly explanation of a plan.
plan_pipeline(pipeline_cls, *, context=None, profile=None, selection=None)
¶
Plan a pipeline. Raises PipelineValidationError when invalid.
plan_pipeline_with_report(pipeline_cls, *, context=None, profile=None, selection=None)
¶
Plan a pipeline, returning (plan, report). Plan is None when invalid.
canonical_plan_json(plan)
¶
Return canonical JSON bytes as a UTF-8 string.
plan_fingerprint(plan)
¶
Compute a stable SHA-256 fingerprint of the canonical plan.
plan_from_json(text, *, verify=True)
¶
Deserialize a plan from JSON text.
When verify is True (default), recompute the fingerprint and reject
tampered plans.
plan_to_json(plan, *, indent=2)
¶
Serialize a plan including its fingerprint.
dependency_closure(graph, targets)
¶
Return declaration-ordered upstream closure for targets (inclusive).
run_one_selection(graph, step_name)
¶
Select a single step and its required upstream closure.
run_until_selection(graph, step_name)
¶
Select declaration-order prefix through step_name (inclusive).
Includes parallel siblings declared earlier than the target, not only the upstream dependency closure.
etlantic.registry
¶
Scoped registries for plugins, implementations, bindings, and providers.
Registries belong to a PlanningContext instance (ADR-004), never process globals.
PluginDescriptor
dataclass
¶
Installed plugin metadata for planning (no live handles).
to_dict()
¶
Serialize plugin descriptor.
ImplementationDescriptor
dataclass
¶
BindingDescriptor
dataclass
¶
RegistryBundle
dataclass
¶
PlanningContext
dataclass
¶
Scoped planning inputs: profile + registries (no live resources).
create(profile=None, *, registry=None, required_capabilities=None, allow_capability_fallback=False)
classmethod
¶
Build a planning context from a profile name/object.
When dataframe_engine is polars or pandas and no custom
registry is supplied, discovered entry-point plugins are registered
onto a stub registry so plan-only paths work without a runtime.
When sql_engine is sql, discovered SQL plugins are registered
the same way. When spark_engine is pyspark/spark,
discovered Spark plugins are registered the same way.
builtin_stub_registry()
¶
Return a registry with in-tree stub plugins for local planning tests.
local is the in-process Python-records path (not a dataframe engine).
Polars/Pandas plugins register themselves via entry points or explicit
register_plugin calls when installed.
etlantic.plugin_trust
¶
Plugin allowlist / version-pin enforcement (0.9).
plugin_allowed(*, name, version, allowlist)
¶
Return True when name is permitted by allowlist (and pin).
filter_plugins_by_allowlist(plugins, profile, *, name_attr='name', version_attr='version')
¶
Filter discovered plugins using profile allowlist.
Production profiles fail closed when the allowlist is empty or a plugin is not listed / does not match the version pin. Non-production profiles with an empty allowlist remain unrestricted.
assert_plugin_trust(plugins, profile)
¶
Filter plugins and raise when production trust fails closed.
etlantic.model
¶
Immutable logical graph intermediate representation.
NodeKind
¶
Bases: StrEnum
Kinds of nodes in a logical pipeline graph.
PortSpec
dataclass
¶
A typed port on a logical node.
ParameterSpec
dataclass
¶
A typed parameter on a transformation step.
Edge
dataclass
¶
A data-flow edge from a producer port to a consumer port.
Node
dataclass
¶
A node in the logical pipeline graph.
LogicalGraph
dataclass
¶
Local runtime and reports¶
etlantic.runtime
¶
Local runtime package.
CancellationPolicy
dataclass
¶
Cancellation behavior.
InvalidationMode
¶
Bases: StrEnum
How prior artifacts are invalidated for a rerun.
MaterializationPolicy
¶
Bases: StrEnum
How intermediate artifacts should be materialized for a run.
RetryPolicy
dataclass
¶
Retry behavior for steps and runs.
RunIntent
¶
Bases: StrEnum
Why a pipeline run was requested.
RunRequest
dataclass
¶
Portable request describing how to execute a pipeline.
asset_overrides
property
¶
Preferred public view of node → logical asset overrides.
RunSelection
dataclass
¶
TimeoutPolicy
dataclass
¶
Timeout behavior for runs and steps.
FailureStage
¶
Bases: StrEnum
Where a node failure occurred.
RunStatus
¶
Bases: StrEnum
Normalized pipeline run status.
StepStatus
¶
Bases: StrEnum
Normalized step / node execution status.
etlantic.lifecycle
¶
Lifecycle extension package.
CallbackRegistry
dataclass
¶
Lifecycle outcome callbacks.
FailureAction
¶
Bases: StrEnum
How the runtime should proceed after a callback.
StepFailureContext
dataclass
¶
Context passed to step-failure callbacks.
MiddlewareStack
dataclass
¶
Deterministic ordered middleware stack.
Emit
dataclass
¶
Bases: Generic[T]
Runtime emission of an outbound event (payload must be secret-free).
OutboundEvent
dataclass
¶
Bases: Generic[T]
Declared outbound event that a pipeline may emit.
Inject
dataclass
¶
Annotation marker for hierarchical resource injection.
ResourceManager
dataclass
¶
Scoped resource acquisition with yield cleanup exactly once.
PipelineRuntime
dataclass
¶
Process-scoped runtime coordinating local execution.
register_dataframe_plugin(engine, plugin)
¶
Register a live dataframe plugin and its planning descriptor.
register_sql_plugin(engine, plugin)
¶
Register a live SQL plugin and its planning descriptor.
register_spark_plugin(engine, plugin)
¶
Register a live Spark plugin and its planning descriptor.
register_spark_provider(name, provider)
¶
Register a live Spark session provider.
register_orchestrator_plugin(engine, plugin)
¶
Register a live orchestrator plugin and its planning descriptor.
apply_plugin_allowlist(profile)
¶
Filter discovered plugins using profile.plugin_allowlist.
Production profiles fail closed. Returns trust diagnostics.
session()
async
¶
Enter runtime lifespan (if any).
etlantic.reports
¶
Run report package.
ArtifactResult
dataclass
¶
Artifact produced or reused during a run.
BackendRunReference
dataclass
¶
Reference to a backend-specific run identity.
PipelineRunReport
dataclass
¶
Canonical secret-free summary of a pipeline run.
from_dict(data)
classmethod
¶
Deserialize a run report from its JSON-friendly dict form.
RunDiagnostic
dataclass
¶
Execution diagnostic (secret-free).
RunRecommendation
dataclass
¶
Suggested follow-up action after a run.
RunSummary
dataclass
¶
Aggregate counters for a run.
SchemaObservationResult
dataclass
¶
Schema observation recorded during a run.
StateTransitionResult
dataclass
¶
Recorded state transition.
StepRunReport
dataclass
¶
Per-step execution summary (secret-free).
ValidationResult
dataclass
¶
Runtime validation outcome at a boundary.
ReportStore
dataclass
¶
Process-local store of completed/partial run reports.
render_html(report)
¶
Render a minimal HTML report (secret-free).
render_text(report)
¶
Render a compact human-readable report.
Storage and secrets¶
etlantic.storage
¶
Local storage bindings for ETLantic 0.4.
CallableStorage
¶
User-registered Python readers/writers keyed by binding name.
CsvStorage
¶
Read/write CSV files using ContractModel field order when available.
JsonStorage
¶
Read/write JSON arrays or JSON Lines files.
MemoryStorage
¶
Named in-process datasets keyed by binding or location.
seed(binding, data, *, location=None)
¶
Seed data for tests and callable pipelines.
NullStorage
¶
Read empty datasets; discard writes (VALIDATE / no-write intents).
StorageBinding
¶
Bases: Protocol
Read/write datasets for Source and Sink nodes.
as_records(data, contract_type)
¶
Normalize data to a list of contract instances or mappings.
records_to_dicts(data)
¶
Convert records to plain dicts for file writers.
etlantic.secrets
¶
Runtime secret resolution package.
SecretCache
dataclass
¶
Process-local bounded secret cache (never serialized).
revoke(*, provider=None, name=None)
¶
Invalidate matching entries; return count removed.
EnvSecretProvider
¶
Resolve secrets from process environment variables.
Looks up {name} or {name}_{key} when key is not the default.
Fail-closed: missing values raise; no silent empty fallback.
MountedFileSecretProvider
¶
Resolve secrets from files under a mount root.
Default path: {root}/{name} or {root}/{name}/{key}.
Fail-closed on missing/unreadable files.
ProviderContext
dataclass
¶
Context for provider lifespan.
SecretProvider
¶
Bases: Protocol
Protocol for runtime secret resolution.
SecretProviderCapabilities
dataclass
¶
Declared secret-provider capabilities.
SecretProviderDescriptor
dataclass
¶
Installed secret provider metadata.
SecretResolutionContext
dataclass
¶
Caller identity for a secret resolution (no values).
SecretRef
dataclass
¶
SecretSerializationError
¶
Bases: TypeError
Raised when a SecretValue is asked to serialize.
SecretValue
dataclass
¶
Runtime-only secret payload with redacted display.
Must never appear in plans, reports, events, or logs. Prefer passing this only to the declared resource consumer.
Contract interchange¶
etlantic.interchange
¶
Contract interchange: ODCS, DTCS, DPCS, bundles, and diffs.
ContractBundle
dataclass
¶
In-memory view of a generated or loaded contract bundle.
ArtifactProvenance
dataclass
¶
Origin metadata for a loaded or generated artifact.
ProvenanceKind
¶
Bases: StrEnum
How a logical artifact was obtained.
generate_contracts(pipeline_cls)
¶
Discover contracts and build an in-memory bundle for pipeline_cls.
load_bundle(directory)
¶
Load a contract bundle directory and reconstruct the pipeline class.
write_contracts(pipeline_cls, directory)
¶
Generate and write a deterministic contract bundle under directory.
diff_data_contracts(previous, current, *, mode=CompatibilityMode.BACKWARD)
¶
Compare two data contracts via ContractModel and return diagnostics.
diff_pipelines(previous, current)
¶
Compare two pipelines / DPCS docs via the dpcs toolkit.
diff_transformations(previous, current)
¶
Compare two transformations / DTCS docs via the dtcs toolkit.
pipeline_from_dpcs(source, *, registry=None, root=None, class_name=None)
¶
Load a DPCS artifact into a dynamic Pipeline subclass.
pipeline_to_dpcs(pipeline_cls, *, data_locations=None, transform_locations=None)
¶
Build a DPCS document dict from a Pipeline subclass.
write_dpcs(pipeline_cls, path, **kwargs)
¶
Write a Pipeline to a DPCS YAML file.
transformation_from_dtcs(source, *, contracts=None, root=None, class_name=None)
¶
Load a DTCS artifact into a dynamic Transformation subclass.
transformation_to_dtcs(transformation_cls)
¶
Build a DTCS document dict from a Transformation subclass.
write_dtcs(transformation_cls, path)
¶
Write a Transformation to a DTCS YAML file.
graphs_equivalent(left, right)
¶
Return True when two logical graphs match by fingerprint.
normalize_pipeline(pipeline_cls)
¶
Return the logical graph and provenance for a pipeline class.
load_data_contract(path, *, root=None, class_name=None)
¶
Load an ODCS file into a ContractModel subclass via ContractModel.
write_odcs(model, path, *, root=None)
¶
Write a ContractModel class to an ODCS YAML file.
Dataframe protocol¶
etlantic.dataframe
¶
Versioned dataframe execution protocol (etlantic.dataframe/1).
Core stays engine-free. Polars/Pandas plugins implement DataframePlugin.
ArtifactOwnership
¶
Bases: StrEnum
Ownership state for a dataframe artifact handle.
DataframeExecutionContext
dataclass
¶
Caller identity and plan-resolved settings for one step invocation.
DataframeMetrics
dataclass
¶
Structured metrics emitted by a dataframe step.
DataframeOutputBundle
dataclass
¶
Normalized outputs from a dataframe plugin invocation.
DataframePhase
¶
Bases: StrEnum
Reportable phases of a dataframe step execution.
DataframePlugin
¶
Bases: Protocol
Protocol for in-process dataframe execution backends.
materialize_input(value, *, contract_type, context, port_name)
¶
Convert a logical/runtime value into a native frame for this engine.
invoke(*, callable_, inputs, parameters, context)
¶
Invoke a registered transformation implementation.
normalize_output(result, *, output_ports, context)
¶
Normalize a callable result into valid/invalid/side outputs.
validate_frame(value, *, contract_type, context, boundary, port_name=None)
¶
Validate a frame (or records) against a contract.
Returns (value, decision, diagnostics, invalid_value).
invalid_value is set when reject/quarantine splits rows.
inspect_schema(value, *, identity)
¶
Return a NormalizedSchema-compatible mapping, if inspectable.
ensure_ownership(value, *, ownership, context)
¶
Return a handle that respects ownership / mutation isolation.
collect_if_needed(value, *, context)
¶
Collect lazy values when the plan declares a collection boundary.
to_records(value, *, contract_type)
¶
Convert a native frame to Python records for storage/publication.
row_count(value)
¶
Best-effort row count without forcing a full materialization when possible.
DataframePluginInfo
dataclass
¶
Installed dataframe plugin metadata.
DataframeValidationOutcome
¶
Bases: StrEnum
Configured outcome when invalid rows are identified.
DataframeValidationPolicy
dataclass
¶
Input/output validation policy for a dataframe step.
ValidationDecision
¶
Bases: StrEnum
Decision applied during a validation phase.
arrow_available()
¶
Return True when pyarrow can be imported.
from_arrow_table(table, *, engine)
¶
Convert an Arrow table into records (core) or raise for engine frames.
Core keeps Arrow↔records only. Engine-native construction belongs in
dataframe plugins (pl.from_arrow / Table.to_pandas). Pass
engine="records" (or any non-plugin name) to get a pylist.
records_to_arrow_table(records, *, contract_type=None)
¶
Build a pyarrow.Table from Python records.
Raises ImportError when PyArrow is not installed.
to_arrow_table(value)
¶
Best-effort conversion of a native frame to pyarrow.Table.
Returns None when the value cannot be converted without engine plugins.
discover_dataframe_plugins()
¶
Load dataframe plugins registered under the entry-point group.
Returns a mapping of engine name → plugin instance. Missing or broken entry points are skipped with a warning (callers should fail closed at planning when the selected engine is absent).
register_discovered_plugins(registry, *, plugins=None)
¶
Register discovered dataframe plugins into a planning registry.
Returns the plugin instances (live handles for the runtime; descriptors only are stored on the registry).
SQL protocol¶
etlantic.sql
¶
Versioned SQL execution protocol (etlantic.sql/1).
Core stays driver-free. Install etlantic-sql for the PostgreSQL
reference plugin.
AliasedExpr
dataclass
¶
Projection expression with output alias.
AtomicPublicationStrategy
¶
Bases: StrEnum
How replace/snapshot must publish atomically.
BinaryExpr
dataclass
¶
Binary expression over portable operators.
CallExpr
dataclass
¶
Function call expression (scalar or aggregate).
CaseWhenExpr
dataclass
¶
Searched CASE expression.
ColumnRef
dataclass
¶
Column reference within a relation alias.
CompiledSql
dataclass
¶
Compiled statement with secret-free parameter metadata.
ConcatExpr
dataclass
¶
String concatenation of columns / literals.
CteDef
dataclass
¶
Common table expression wrapping a subquery.
JoinClause
dataclass
¶
JOIN clause against another relation (fail-collision policy only).
LiteralExpr
dataclass
¶
Typed literal (serialized into bound parameters, never interpolated).
OrderByItem
dataclass
¶
ORDER BY key with null placement.
RelationRef
dataclass
¶
Logical relation identity without credentials or live connections.
parse(value)
classmethod
¶
Parse catalog.schema.name or schema.name or name.
SqlExecutionContext
dataclass
¶
Runtime identity and connection binding for SQL work.
SqlExecutionResult
dataclass
¶
Result of compiling/executing SQL work.
SqlMetrics
dataclass
¶
Normalized SQL execution metrics (no query values).
SqlParameter
dataclass
¶
Named parameter placeholder (value resolved only at runtime).
SqlPhase
¶
Bases: StrEnum
Reportable phases of SQL region / step execution.
SqlPlugin
¶
Bases: Protocol
Protocol for SQL execution backends.
capabilities()
¶
Return declared SQL / dialect capabilities.
quote_identifier(name)
¶
Validate and quote an identifier; raise on illegal names.
relation_from_binding(*, binding, location, metadata=None)
¶
Map a storage/binding location to a RelationRef.
compile_query(query, *, context)
¶
Compile a portable query to parameterized dialect SQL.
compile_write(write, *, context)
¶
Compile a write / publication intent.
execute(compiled, *, params, context, fetch=False)
¶
Execute compiled statements.
When fetch is False (SQL-to-SQL), must not materialize rows into
Python. When True (hybrid boundary), may return records.
execute_write(write, *, params, context)
¶
Compile and execute a write intent.
materialize_temp(query, *, temp_name, params, context)
¶
Create a temporary relation from a query without fetching rows.
load_records(records, *, target, context)
¶
Load Python/dataframe records into a staging relation.
fetch_records(relation, *, params, context, contract_type=None)
¶
Fetch rows at a hybrid boundary (counts toward rows_fetched).
inspect_relation(relation, *, context)
¶
Inspect catalog metadata without reading source rows.
rows_fetched_total()
¶
Instrumentation: cumulative rows fetched into Python.
SqlPluginInfo
dataclass
¶
Installed SQL plugin metadata.
SqlQuery
dataclass
¶
Closed portable select query IR (kernel + relational /1 surface).
SqlWrite
dataclass
¶
Write / publication intent against a target relation.
TransactionOutcome
¶
Bases: StrEnum
Known transaction outcomes for retry gating.
TrustedSqlFragment
dataclass
¶
Escape hatch for trusted SQL text (policy-gated; disabled by default).
UnaryExpr
dataclass
¶
Unary expression (NOT, negate).
WriteIntentKind
¶
Bases: StrEnum
Portable SQL write / publication intents.
discover_sql_plugins()
¶
Load SQL plugins registered under the entry-point group.
load_sql_plugin(engine='sql')
¶
Return a discovered plugin for engine, or None.
register_discovered_plugins(registry, *, plugins=None)
¶
Register discovered SQL plugins into a planning registry.
alias(expr, name)
¶
Alias a projection expression.
col(name, relation=None)
¶
Column reference.
concat(*parts, separator=' ', as_=None)
¶
Concatenate string parts (columns or literals).
lit(value, *, sql_type=None)
¶
Literal that will be bound as a parameter.
select(*columns, source, where=None, limit=None, distinct=False, parameters=())
¶
Build a portable select query.
trusted_sql(text, *, param_names=(), allowed=False)
¶
Trusted SQL fragment; allowed must be True under profile policy.
is_safe_identifier(name)
¶
Return True when name is a simple unquoted-safe identifier.
redact_params(names)
¶
Build redacted parameter metadata for plans/logs.
require_safe_identifier(name)
¶
Validate identifier; raise ValueError when unsafe.
sqlmodel_table_to_relation(table)
¶
Optional SQLModel table → RelationRef + schema metadata.
Imports SQLModel only when called. Core never depends on SQLModel.
Spark protocol¶
etlantic.spark
¶
Versioned Spark execution protocol (etlantic.spark/1).
Core stays PySpark-free. Install etlantic-pyspark for the reference plugin.
Streaming APIs are experimental in 0.7.
CompiledSparkPlan
dataclass
¶
Compiled Spark region — secret-free and inspectable.
DatasetRef
dataclass
¶
Logical Spark dataset identity without credentials or live sessions.
ExpressionStrategy
¶
Bases: StrEnum
How a compiled step realizes expressions.
SchemaCompatibility
¶
Bases: StrEnum
Contract ↔ Spark schema mapping outcome (never guess).
SparkAction
dataclass
¶
An explicit action boundary recorded in compiled plans.
SparkActionKind
¶
Bases: StrEnum
Explicit Spark actions (no hidden collect/show/toPandas).
SparkCompilationContext
dataclass
¶
Inputs for region compilation (no live session).
SparkDataFrameHandle
dataclass
¶
Portable handle wrapping a native Spark DataFrame identity.
The live frame lives only inside the plugin/runtime; plans and reports never embed sessions or credentials.
SparkExecutionContext
dataclass
¶
Runtime context for Spark execution.
SparkExecutionResult
dataclass
¶
Result of executing a compiled Spark plan or step.
SparkMetrics
dataclass
¶
Normalized Spark execution metrics (no secret values).
SparkPhase
¶
Bases: StrEnum
Reportable phases of Spark region / step execution.
SparkPlanRegion
dataclass
¶
Planner-facing description of a Spark execution region.
SparkPlugin
¶
Bases: Protocol
PySpark / Spark execution plugin protocol.
SparkPluginInfo
dataclass
¶
Discoverable plugin metadata.
SparkUdfPolicy
¶
Bases: StrEnum
Profile policy for UDF usage during planning/compile.
SparkWrite
dataclass
¶
Portable write intent realized by the Spark/Delta plugin.
SparkWriteMode
¶
Bases: StrEnum
Delta-compatible portable write modes.
ResourceContext
dataclass
¶
Runtime context for acquire/release (may carry secret resolvers).
SessionOwnership
¶
Bases: StrEnum
Who owns the SparkSession lifecycle.
SparkProvider
¶
Bases: Protocol
Supplies SparkSession lifecycle without embedding secrets in plans.
SparkProviderInfo
dataclass
¶
Discoverable provider metadata.
SparkSessionHandle
dataclass
¶
Opaque handle for an acquired session (no credentials).
SparkSessionRequest
dataclass
¶
Secret-free request for a Spark session.
Credentials and secret values are resolved by the provider at acquire time
via secret_refs / runtime secret providers — never embedded here.
FieldMapping
dataclass
¶
One field mapping outcome.
SchemaMappingResult
dataclass
¶
Aggregate contract ↔ Spark schema mapping.
LateEventPolicy
¶
Bases: StrEnum
Policy for events past the watermark.
StreamingOutputMode
¶
Bases: StrEnum
Streaming output modes.
StreamingProgress
dataclass
¶
Normalized streaming progress evidence.
StreamingQuerySpec
dataclass
¶
Experimental streaming query declaration.
StreamingTrigger
¶
Bases: StrEnum
Supported Structured Streaming triggers.
WatermarkSpec
dataclass
¶
Event-time watermark configuration.
discover_spark_plugins()
¶
Load Spark plugins registered under the entry-point group.
discover_spark_providers()
¶
Load Spark providers registered under the entry-point group.
load_spark_plugin(engine='pyspark')
¶
Return a discovered plugin for engine, or None.
load_spark_provider(name='local')
¶
Return a discovered provider by name, or None.
register_discovered_plugins(registry, *, plugins=None)
¶
Register discovered Spark plugins into a planning registry.
compare_types(logical, spark_type)
¶
Compare logical vs observed Spark type without guessing.
map_contract_schema(contract_type, *, observed=None)
¶
Map a ContractModel / Pydantic model to Spark types with explicit outcomes.
observation_from_spark_schema(spark_schema, *, source='spark', partition_columns=None)
¶
Build a normalized schema observation from a Spark StructType-like object.
Orchestration protocol¶
etlantic.orchestration
¶
External orchestration protocols (ETLantic 0.8).
ArtifactTransportPolicy
dataclass
¶
Policy controlling how step outputs cross orchestrator task boundaries.
OrchestrationCompilationError
¶
CancellationResult
dataclass
¶
Result of requesting cancellation of an external run.
CorrelationKeys
dataclass
¶
Keys linking ETLantic runs to backend orchestrator identities.
PollResult
dataclass
¶
Polled status for an external orchestration run.
SubmissionResult
dataclass
¶
Result of submitting a compiled artifact to an orchestrator.
SubmissionStatus
¶
Bases: StrEnum
Lifecycle of an externally submitted orchestration run.
CompilationContext
dataclass
¶
Context supplied to orchestrator compilers.
CompilationDiagnostic
dataclass
¶
Structured diagnostic from orchestration compilation.
CompiledOrchestrationArtifact
dataclass
¶
Deterministic compiled orchestration artifact (secret-free).
write(path)
¶
Write the generated module source to path.
CompiledTask
dataclass
¶
Backend-agnostic compiled task derived from a plan node.
ExecutionIntent
dataclass
¶
Portable execution settings for orchestration backends.
OrchestrationPhase
¶
Bases: StrEnum
Phases of external orchestration.
OrchestratorPlugin
¶
Bases: Protocol
Protocol for external orchestration compilers.
OrchestratorPluginInfo
dataclass
¶
Identity and capability declaration for an orchestrator plugin.
ScheduleIntent
dataclass
¶
Portable scheduling intent (orchestrator maps to backend constructs).
TaskRetryPolicy
¶
Bases: StrEnum
How retries are applied on a compiled task.
validate_artifact_for_transport(artifact, *, estimated_bytes=None, policy=None)
¶
Return diagnostics when an artifact cannot safely cross task boundaries.
xcom_safe_payload(artifact)
¶
Return a small, secret-free payload suitable for XCom / task messaging.
compile_plan(plan, *, target='airflow', profile=None, plugin=None, context=None, plugins=None)
¶
Compile a secret-free PipelinePlan into an orchestration artifact.
Fails closed when the target plugin is missing or cannot preserve required semantics.
explain_compilation(artifact)
¶
Return a JSON-friendly compilation explanation.
discover_orchestrator_plugins()
¶
Load orchestrator plugins registered under the entry-point group.
load_orchestrator_plugin(engine='airflow')
¶
Return a discovered orchestrator plugin for engine, or None.
plugin_registry_snapshot()
¶
Return serializable descriptors for discovered orchestrator plugins.
register_discovered_plugins(registry, *, plugins=None)
¶
Register discovered orchestrator plugins into a planning registry.
comparable_report_shape(report)
¶
Normalized shape for comparing local vs orchestrated runs.
correlate_poll_to_report(report, poll)
¶
Attach backend correlation and task states onto a normalized report.
context_from_profile(profile, *, target, max_inline_bytes=65536)
¶
Build a compilation context from a profile.
dag_id_for_plan(plan)
¶
Deterministic DAG / workflow id from plan identity.
execution_from_profile(profile)
¶
Extract portable execution intent from profile fields/metadata.
map_plan_to_tasks(plan, *, context)
¶
Map logical graph nodes/edges to a compiled task graph.
schedule_from_profile(profile)
¶
Extract portable schedule intent from profile fields/metadata.
apply_reliability_to_task(task, *, plan, execution)
¶
Return an updated task with retry + reliability fields applied.
reliability_metadata(plan, node_name)
¶
Collect portable repair/backfill/reconciliation fields for a node.
resolve_task_retry(*, node_name, node_kind, plan, execution, declarations=None)
¶
Map profile retry intent to a safe per-task policy.
Unsafe sinks with requested retries produce PMORCH310 and force retries
off (or reject when strict metadata asks).
Visualization¶
etlantic.viz
¶
Shared graph IR and visualization exporters beyond Mermaid (0.9).
Public helpers:
logical_graph_to_ir/plan_to_ir— build aGraphIRgraph_to_dot— Graphviz DOT textgraph_to_html— single-page HTML lineage documentlineage_export— JSON lineage document (etlantic.lineage/1)
CLI: etlantic viz dot|html|lineage.
GraphIR
dataclass
¶
Backend-neutral graph intermediate representation.
graph_to_dot(ir)
¶
Export Graphviz DOT (no graphviz binary required to generate text).
ir_to_logical_graph(ir)
¶
Rebuild a LogicalGraph suitable for Mermaid rendering.
graph_to_html(ir, *, include_mermaid=True)
¶
Simple HTML lineage/docs page (stdlib only).
lineage_export(ir)
¶
JSON lineage document suitable for docs / agents.
etlantic.mermaid
¶
Mermaid diagram generation from logical graphs.
graph_to_mermaid(graph)
¶
Render a logical pipeline graph as a Mermaid flowchart.
Output is deterministic for a given graph.
Agents, IDE, and notebooks¶
etlantic.agents
¶
Agent guidance generators (AGENTS.md, CLAUDE.md, Codex SKILL, Cursor rules).
generate_agent_guidance(root, *, overwrite=True)
¶
Write agent guidance files under root.
etlantic.ide
¶
etlantic.notebook
¶
Observability¶
etlantic.observability
¶
Schema history¶
etlantic.schema_history
¶
File-backed schema history provider (no source rows).
Capabilities¶
etlantic.capabilities
¶
Plugin capability declarations and negotiation results.
CapabilityDecision
¶
Bases: StrEnum
Outcome of comparing required vs available capabilities.
PluginCapabilities
dataclass
¶
Declared capabilities of a plugin or engine.
Dataframe-oriented flags (eager/lazy/arrow/...) are first-class for 0.5
planning. SQL-oriented flags are first-class for 0.6. Unknown requirements
may still be declared via extras.
CapabilityNegotiation
dataclass
¶
Record of a capability check for one requirement.
to_dict()
¶
Serialize negotiation record.
negotiate_capabilities(*, requirements, available, fallback=None, allow_fallback=False)
¶
Negotiate required capabilities against an available engine.
Unsupported requirements fail closed unless allow_fallback is True and
a fallback engine covers the requirement.
Reliability and schema drift¶
etlantic.reliability
¶
Portable reliability and intent models (schemas in 0.3; enforcement in 0.4+).
WriteMode
¶
Bases: StrEnum
Declared write semantics.
MaterializationMode
¶
Bases: StrEnum
Declared materialization intent.
FreshnessExpectation
dataclass
¶
PartitionCompletenessExpectation
dataclass
¶
WriteIntent
dataclass
¶
MaterializationIntent
dataclass
¶
IdempotencyDeclaration
dataclass
¶
RetrySafetyDeclaration
dataclass
¶
ReconciliationDeclaration
dataclass
¶
BackfillDeclaration
dataclass
¶
RepairDeclaration
dataclass
¶
ReliabilityEvidence
dataclass
¶
environment_identity(*, profile, security_domain, workspace='')
¶
Deterministic identity for a resolved environment.
implementation_selection_identity(*, transformation_id, engine)
¶
Deterministic identity for a selected implementation.
quality_metric_identity(*, subject_id, metric)
¶
Deterministic identity for a quality metric.
statistical_observation_identity(*, subject_id, feature)
¶
Deterministic identity for a statistical observation.
fingerprint_mapping(data)
¶
Deterministic fingerprint for a mapping.
etlantic.schema_drift
¶
Schema drift models: normalized schema, observations, changes, impact (0.3).
DriftImpact
¶
Bases: StrEnum
Impact vocabulary for schema changes.
NormalizedSchema
dataclass
¶
SchemaObservation
dataclass
¶
Immutable schema observation (no row data or secrets).
to_dict()
¶
Serialize observation.
normalize_schema_from_model(model, *, identity=None)
¶
Normalize a ContractModel/Data class into a logical schema.
normalize_schema_from_fields(fields, *, identity)
¶
Normalize a list of field mappings (operational observation path).
diff_normalized_schemas(baseline, candidate)
¶
Operational drift path: compare normalized schemas without ContractModel.
diff_contract_schemas(previous, current)
¶
Contract-drift path: delegate compatibility meaning to ContractModel diffs.
etlantic.schema_policy
¶
Schema drift policy decisions for local runtime.
Testing helpers¶
etlantic.testing
¶
ETLantic testing helpers.
run_conformance_suite(plugin, *, engine, sample_rows, contract_type=None)
¶
Minimal conformance checks for third-party dataframe plugins.
run_orchestrator_conformance_suite(plugin, *, engine, plan, context=None)
¶
Compile a plan and assert a secret-free artifact is produced.
normalize_rows(rows)
¶
Stable cross-engine row normalization for conformance compares.
run_portable_transform_conformance_suite(compiler, *, profiles=None, to_frame=None, enforce_fixture_coverage=True)
¶
Run capability-selected portable transform conformance for compiler.
The suite selects mandatory fixtures for every advertised profile/action/
function claim. Compilers must pass every selected fixture (or correctly
reject unsupported modes at analyze time). When
enforce_fixture_coverage is true, claiming kernel/relational profiles
or individual actions/functions without a matching fixture fails the suite.
assert_missing_secret_fails(provider, reference)
¶
Providers must fail closed when a secret is absent.
run_secret_conformance_suite(provider, *, reference, expected_substring=None)
¶
Resolve one secret and assert SecretValue hygiene.
run_sql_conformance_suite(plugin)
¶
Minimal conformance checks for SQL plugins (driver-backed).
assert_write_intent_parity(backends, *, subject_id, mode)
¶
Assert each backend can declare the same write intent vocabulary.
backends maps engine name → object with supports_write_mode(mode)
or a capabilities mapping containing write mode strings.
run_write_semantics_parity_suite(backends, *, subject_id='parity', modes=(WriteMode.APPEND, WriteMode.OVERWRITE))
¶
Run parity checks for shared write intents across backends.
Exceptions¶
etlantic.exceptions
¶
Public exception hierarchy for ETLantic.
ETLanticError
¶
Bases: Exception
Base class for public ETLantic exceptions.
ModelDefinitionError
¶
Bases: ETLanticError
Raised when a class definition cannot form a usable model.
PipelineValidationError
¶
InternalETLanticError
¶
Bases: ETLanticError
Raised when an internal invariant is violated.
PipelineExecutionError
¶
NodeExecutionError
¶
DataValidationError
¶
PipelineTimeoutError
¶
PipelineCancelledError
¶
Stability¶
ETLantic is alpha. A root export is public in the current release, but 0.x releases may change APIs. Review the changelog and compatibility matrix before upgrading. Narrative CLI docs: CLI.