API — Plan and Runtime¶

Generated from package source. Hub: Python API Reference.

0.21 trust and plan essentials¶

API	Behavior
`Profile.security_mode`	`development` \| `test` \| `production`; production fail-closed trust uses mode only
`Profile.plugin_allowlist`	Required in production; evaluated before plugin import
`Profile.safe_io` / `Profile.outbound`	Safe filesystem writes and outbound HTTP policy (0.20)
`resolve_profile(name, allow_adhoc_profile=False)`	Unknown bare names raise `PMCFG100` unless ad hoc is allowed
`Profile.from_dict(..., accept_legacy_bindings=False)`	Legacy `bindings`-only JSON fails closed with `PMCFG111`; pass `True` / `--accept-legacy-bindings` to allow
`PipelinePlan.from_dict` / `plan_from_json`	Require wire `schema: "etlantic.plan/1"`; verify fingerprint by default
`verify_plan_fingerprint(plan)`	Public check; also called before `compile_plan` and local run
`deep_freeze(value)`	Recursively freeze plan-owned nests

See Migration 0.20 → 0.21 and What's new in 0.21.

Validation and diagnostics¶

`etlantic.diagnostics` ¶

Structured diagnostics and validation reports.

`Severity` ¶

Bases: StrEnum

Diagnostic severity levels.

`SourceLocation` `dataclass` ¶

Optional origin of a diagnostic (file, Python object, or contract path).

`DiagnosticAction` `dataclass` ¶

Machine-readable action a future IDE can expose as a quick fix.

`to_dict()` ¶

Serialize action for tooling.

`Diagnostic` `dataclass` ¶

A structured finding from loading, inspection, or validation.

`ValidationReport` `dataclass` ¶

Immutable collection of diagnostics from a validation pass.

Produced by :meth:~etlantic.pipeline.Pipeline.validate and related loaders. Use :meth:raise_for_errors to fail closed in application or CI code. Serialize to JSON or SARIF via CLI etlantic validate --format.

`valid` `property` ¶

Return True when no error-severity diagnostics are present.

`has_errors` `property` ¶

Return True when at least one error-severity diagnostic is present.

`errors` `property` ¶

Return error-severity diagnostics.

`warnings` `property` ¶

Return warning-severity diagnostics.

`filter(*, severity=None, code=None, phase=None)` ¶

Return a report containing only matching diagnostics.

`raise_for_errors()` ¶

Raise :class:PipelineValidationError if the report is invalid.

`merge(other)` ¶

Combine two reports, preserving order and uniqueness by identity.

`from_diagnostics(diagnostics, *, phases=())` `classmethod` ¶

Build a report from an iterable of diagnostics.

`codes()` ¶

Return diagnostic codes in report order.

`etlantic.validation` ¶

Multi-phase validation for ETLantic pipelines (0.3).

`validate_pipeline(pipeline_cls, *, context=None, profile=None, policy=None)` ¶

Validate a pipeline through structural → capability phases.

`validate_transformation(transform)` ¶

Validate a transformation class definition in isolation.

`etlantic.policy` ¶

Named validation and quality-gate policies.

`PolicyMode` ¶

Bases: StrEnum

How a policy treats findings.

`ValidationPolicy` `dataclass` ¶

Named validation / quality-gate policy selected by profile or caller.

`to_dict()` ¶

Serialize policy.

`register_validation_policy(policy)` ¶

Register a named validation policy for later resolve_validation_policy.

`resolve_validation_policy(name)` ¶

Resolve a policy name or object.

Unknown names fail closed rather than inventing an empty default policy.

Profiles, planning, and registries¶

`etlantic.profile` ¶

Execution profiles that bind logical pipelines to environments.

`Profile` `dataclass` ¶

Environment binding for planning and execution without changing logic.

A profile selects engines (local Python, Polars, SQL, PySpark, …), maps logical asset names to storage providers, and carries trust and policy settings. The same :class:~etlantic.pipeline.Pipeline class can be validated, planned, and run under different profiles.

Important fields:

assets (public) / internal bindings: logical asset → provider id
dataframe_engine, sql_engine, spark_engine: engine selection
security_mode: development, test, or production
plugin_allowlist: production fail-closed plugin trust (name → pin)
portable_transform_policy: prefer, require, or native
safe_io, outbound: 0.20 safe I/O and outbound HTTP policy

Production profiles with security_mode="production" require a non-empty plugin_allowlist and readable static plugin manifests before import.

Use :func:development_profile, :func:test_profile, or :func:production_profile as templates, or resolve names and JSON paths with :func:resolve_profile.

`engine_profile` `property` ¶

Internal engine configuration view.

`assets` `property` ¶

Preferred public view of logical asset → provider resolution.

`primary_engine()` ¶

Resolve the profile's primary execution engine (spark → sql → dataframe).

`identity()` ¶

Stable profile identity.

`to_dict()` ¶

Serialize profile for public JSON (assets only; no mirrored bindings).

`to_plan_snapshot()` ¶

Fingerprint-stable snapshot retaining the plan wire bindings shape.

`from_plan_snapshot(data)` `classmethod` ¶

Rehydrate a profile embedded in a frozen plan (bindings wire shape).

`from_dict(data, *, accept_legacy_bindings=False)` `classmethod` ¶

Deserialize a profile mapping.

Legacy JSON bindings keys require accept_legacy_bindings=True in 0.21+ (use profile migrate to rewrite to assets).

`with_updates(**kwargs)` ¶

Return a copy with selected fields replaced.

Unknown keys raise TypeError so typos like plugin_allow_list cannot silently leave production allowlists empty. assets is the public authoring key for the internal bindings store.

`development_profile(**overrides)` ¶

Return the built-in development profile template.

Defaults: orchestrator="local", dataframe_engine="local", security_mode="development", validation_policy="default".

Parameters:

Name	Type	Description	Default
`**overrides`	`Any`	Passed to :meth:`Profile.with_updates` when non-empty.	`{}`

Returns:

Type	Description
`Profile`	A concrete :class:`Profile` instance.

`test_profile(**overrides)` ¶

Return the built-in test profile template.

Defaults: security_mode="test", validation_policy="strict".

Parameters:

Name	Type	Description	Default
`**overrides`	`Any`	Passed to :meth:`Profile.with_updates` when non-empty.	`{}`

Returns:

Type	Description
`Profile`	A concrete :class:`Profile` instance.

`production_profile(**overrides)` ¶

Return the built-in production profile template.

Defaults: security_mode="production", validation_policy="strict". Callers must still set plugin_allowlist (and usually assets) before planning or running in production.

Parameters:

Name	Type	Description	Default
`**overrides`	`Any`	Passed to :meth:`Profile.with_updates` when non-empty.	`{}`

Returns:

Type	Description
`Profile`	A concrete :class:`Profile` instance.

`resolve_profile(profile, *, allow_adhoc_profile=False)` ¶

Resolve a profile name, JSON path, or object to a concrete Profile.

Resolution order for string profile:

Existing .json file path → :func:load_profile
Built-in template name (development, local, test, production, prod, dev) → template from :data:PROFILE_TEMPLATES
Unknown bare name → fail closed unless allow_adhoc_profile is True

Parameters:

Name	Type	Description	Default
`profile`	`str \| Profile \| None`	Template name, JSON path, :class:`Profile`, or `None`. `None` resolves to `development_profile(name="local")`.	required
`allow_adhoc_profile`	`bool`	When True, unknown bare names become ad hoc development profiles. CLI exposes this as `--allow-adhoc-profile`.	`False`

Returns:

Type	Description
`Profile`	A concrete :class:`Profile`.

Raises:

Type	Description
`FileNotFoundError`	When `profile` ends with `.json` but the path does not exist.
`ValueError`	When the name is unknown and `allow_adhoc_profile` is False (message includes diagnostic `PMCFG100`).

`write_profile(profile, path)` ¶

Write a profile as JSON through :class:~etlantic.io_policy.SafeIoPolicy.

Parameters:

Name	Type	Description	Default
`profile`	`Profile`	Profile to serialize.	required
`path`	`str \| Path`	Destination `.json` path (parent directories created).	required

Returns:

Type	Description
`Path`	Resolved absolute output path.

`load_profile(path, *, accept_legacy_bindings=False)` ¶

Load a profile from a JSON file.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Profile JSON document.	required

Returns:

Name	Type	Description
`Parsed`	`Profile`	class:`Profile` (legacy `bindings` keys may warn `PMCFG110`).

Raises:

Type	Description
`ValueError`	When the document is not a JSON object or fields are invalid.
`OSError`	When the file cannot be read.
`UnsafeLoadError`	When the path escapes the profile directory root.

`etlantic.plan` ¶

PipelinePlan IR package.

`ArtifactRef` `dataclass` ¶

Runtime/durable artifact identity (secret-free).

`to_dict()` ¶

Serialize artifact reference.

`from_dict(data)` `classmethod` ¶

Deserialize artifact reference.

`ArtifactStrategy` ¶

Bases: StrEnum

How a logical OutputRef is realized for consumers.

`PipelinePlan` `dataclass` ¶

Immutable, versioned, secret-free execution-facing intermediate representation.

Produced by :func:~etlantic.plan.planner.plan_pipeline or :meth:~etlantic.pipeline.Pipeline.plan. Wire schema id: :data:~etlantic.plan.model.PLAN_SCHEMA (etlantic.plan/1).

Plans contain secret references only — never resolved secret values. Nested mappings owned by the plan are deep-frozen after construction.

Serialize with :meth:to_dict / :func:~etlantic.plan.serialize.plan_to_json. Deserialize with :meth:from_dict / :func:~etlantic.plan.serialize.plan_from_json. Verify integrity with :func:~etlantic.plan.serialize.verify_plan_fingerprint before compile or run trust boundaries.

`to_dict()` ¶

Serialize plan to a JSON-friendly dict.

`compile(*, target='airflow', profile=None, plugin=None, **kwargs)` ¶

Compile this plan for an external orchestrator.

Delegates to :func:etlantic.orchestration.compile_plan. Verifies the plan fingerprint before compilation.

Parameters:

Name	Type	Description	Default
`target`	`str`	Orchestrator engine id (default `"airflow"`).	`'airflow'`
`profile`	`Any`	Optional profile for compilation context.	`None`
`plugin`	`Any`	Optional pre-constructed orchestrator plugin instance.	`None`
`**kwargs`	`Any`	Forwarded to :func:`~etlantic.orchestration.compile_plan`.	`{}`

Returns:

Type	Description
`Any`	class:`~etlantic.orchestration.protocol.CompiledOrchestrationArtifact`.

Raises:

Type	Description
`ValueError`	When the embedded fingerprint does not match content.
`OrchestrationCompilationError`	When compilation fails closed.

`from_dict(data, *, verify=True)` `classmethod` ¶

Deserialize a plan mapping.

Requires schema equal to :data:PLAN_SCHEMA. Missing or unknown schemas are rejected (no silent default). Documents are upgraded via :func:etlantic.plan.upgrade.upgrade_plan_dict first.

When verify is True and a non-empty fingerprint is present, the embedded fingerprint is checked after construction. Empty fingerprints (e.g. intermediate planner builds) skip verification.

`explain_plan(plan)` ¶

Return a structured, tooling-friendly explanation of a plan.

Suitable for CLI etlantic plan explain, IDE tooling, and debugging. Does not mutate or re-plan the pipeline.

Parameters:

Name	Type	Description	Default
`plan`	`PipelinePlan`	Resolved plan to summarize.	required

Returns:

Type	Description
`dict[str, Any]`	JSON-serializable dict with keys such as `steps`, `regions`,
`dict[str, Any]`	`materialization_boundaries`, `capability_decisions`, and
`dict[str, Any]`	`fingerprint`.

`deep_freeze(value)` ¶

Recursively freeze nested mappings and sequences.

dict / Mapping → MappingProxyType
list → tuple
primitives and frozen dataclass instances are left alone

`validate_plan_interchange(plan)` ¶

Fail closed on invalid tabular interchange boundary metadata.

`plan_pipeline(pipeline_cls, *, context=None, profile=None, selection=None)` ¶

Resolve a validated logical pipeline into a secret-free :class:PipelinePlan.

Validates the pipeline for the resolved profile, then builds the immutable plan IR (schema etlantic.plan/1). No transformation code runs during planning.

Parameters:

Name	Type	Description	Default
`pipeline_cls`	`type[Any]`	Pipeline class to plan.	required
`context`	`PlanningContext \| None`	Optional planning context (registry, profile object, selection). When omitted, `PlanningContext.create(profile=profile)` is used.	`None`
`profile`	`str \| Any \| None`	Profile name, JSON path, :class:`~etlantic.profile.Profile`, or `None` (defaults via :func:`~etlantic.profile.resolve_profile`).	`None`
`selection`	`dict[str, Any] \| None`	Optional partial-run selection (`run_one`, `run_until`).	`None`

Returns:

Type	Description
`PipelinePlan`	Immutable, fingerprinted :class:`~etlantic.plan.model.PipelinePlan`.

Raises:

Type	Description
`PipelineValidationError`	When validation reports errors.

`plan_pipeline_with_report(pipeline_cls, *, context=None, profile=None, selection=None)` ¶

Plan a pipeline and always return the validation report.

Same validation and planning rules as :func:plan_pipeline, but returns (None, report) instead of raising when validation fails.

Parameters:

Name	Type	Description	Default
`pipeline_cls`	`type[Any]`	Pipeline class to plan.	required
`context`	`PlanningContext \| None`	Optional planning context.	`None`
`profile`	`str \| Any \| None`	Profile name, path, object, or `None`.	`None`
`selection`	`dict[str, Any] \| None`	Optional partial-run selection.	`None`

Returns:

Type	Description
`PipelinePlan \| None`	`(plan, report)` where `plan` is `None` when validation failed or
`ValidationReport`	planning raised :class:`~etlantic.exceptions.PipelineValidationError`.

`canonical_plan_json(plan)` ¶

Return canonical JSON bytes as a UTF-8 string.

`plan_fingerprint(plan)` ¶

Compute a stable SHA-256 fingerprint of the canonical plan.

`plan_from_json(text, *, verify=True)` ¶

Deserialize a plan from JSON text.

Parameters:

Name	Type	Description	Default
`text`	`str`	UTF-8 JSON object matching `etlantic.plan/1`.	required
`verify`	`bool`	When True (default), validate wire `schema` and recompute the fingerprint after :meth:`PipelinePlan.from_dict`.	`True`

Returns:

Name	Type	Description
`Parsed`	`PipelinePlan`	class:`~etlantic.plan.model.PipelinePlan`.

Raises:

Type	Description
`ValueError`	When JSON is not an object, schema is missing/unknown, or (when `verify`) the fingerprint does not match content.
`UnsupportedPlanSchemaError`	When the document schema cannot be upgraded.

`plan_to_json(plan, *, indent=2)` ¶

Serialize a plan including its fingerprint.

`verify_plan_fingerprint(plan)` ¶

Recompute the canonical plan fingerprint and compare to plan.fingerprint.

Parameters:

Name	Type	Description	Default
`plan`	`PipelinePlan`	Plan whose embedded fingerprint is checked.	required

Raises:

Type	Description
`ValueError`	When the embedded fingerprint does not match the canonical SHA-256 of the plan content (excluding derived `plan_id` fields).

`dependency_closure(graph, targets)` ¶

Return declaration-ordered upstream closure for targets (inclusive).

`run_one_selection(graph, step_name)` ¶

Select a single step and its required upstream closure.

`run_until_selection(graph, step_name)` ¶

Select declaration-order prefix through step_name (inclusive).

Includes parallel siblings declared earlier than the target, not only the upstream dependency closure.

`etlantic.registry` ¶

Scoped registries for plugins, implementations, bindings, and providers.

Registries belong to a PlanningContext instance (ADR-004), never process globals.

`PluginDescriptor` `dataclass` ¶

Installed plugin metadata for planning (no live handles).

`to_dict()` ¶

Serialize plugin descriptor.

`ImplementationDescriptor` `dataclass` ¶

Resolved implementation selection for a transformation/engine.

`to_dict()` ¶

Serialize implementation descriptor.

`from_dict(data)` `classmethod` ¶

Deserialize implementation descriptor.

`BindingDescriptor` `dataclass` ¶

Logical binding resolved to a provider descriptor (not credentials).

`to_dict()` ¶

Serialize binding descriptor.

`from_dict(data)` `classmethod` ¶

Deserialize binding descriptor.

`RegistryBundle` `dataclass` ¶

Mutable, scoped registries used during planning.

`register_plugin(descriptor)` ¶

Register a plugin descriptor.

`register_binding(descriptor)` ¶

Register a binding descriptor.

`register_implementation(descriptor)` ¶

Register an implementation descriptor keyed by transform+engine.

`PlanningContext` `dataclass` ¶

Scoped planning inputs: profile + registries (no live resources).

`create(profile=None, *, registry=None, required_capabilities=None, allow_capability_fallback=False, allow_adhoc_profile=False)` `classmethod` ¶

Build a planning context from a profile name/object.

When dataframe_engine is polars or pandas and no custom registry is supplied, discovered entry-point plugins are registered onto a stub registry so plan-only paths work without a runtime. When sql_engine is sql, discovered SQL plugins are registered the same way. When spark_engine is pyspark/spark, discovered Spark plugins are registered the same way.

`builtin_stub_registry()` ¶

Return a registry with in-tree stub plugins for local planning tests.

local is the in-process Python-records path (not a dataframe engine). Polars/Pandas plugins register themselves via entry points or explicit register_plugin calls when installed.

`etlantic.plugin_trust` ¶

Plugin allowlist / version-pin enforcement (0.9).

`is_production_profile(profile=None, *, name=None, security_domain=None, security_mode=None)` ¶

Return True when fail-closed production trust/drift applies.

ETLantic 0.19 uses explicit Profile.security_mode == "production" only. name / security_domain remain labels for compatibility and are not used for this decision.

`plugin_allowed(*, name, version, allowlist)` ¶

Return True when name is permitted by allowlist (and pin).

`filter_plugins_by_allowlist(plugins, profile, *, name_attr='name', version_attr='version')` ¶

Filter discovered plugins using profile allowlist.

Production profiles fail closed when the allowlist is empty or a plugin is not listed / does not match the version pin. Non-production profiles with an empty allowlist remain unrestricted.

`assert_plugin_trust(plugins, profile)` ¶

Filter plugins and raise when production trust fails closed.

`etlantic.model` ¶

Immutable logical graph intermediate representation.

`NodeKind` ¶

Bases: StrEnum

Kinds of nodes in a logical pipeline graph.

`PortSpec` `dataclass` ¶

A typed port on a logical node.

`ParameterSpec` `dataclass` ¶

A typed parameter on a transformation step.

`Edge` `dataclass` ¶

A data-flow edge from a producer port to a consumer port.

`Node` `dataclass` ¶

A node in the logical pipeline graph.

`LogicalGraph` `dataclass` ¶

Immutable, deterministic logical pipeline graph.

`node_map()` ¶

Return nodes keyed by name in declaration order.

`node_names()` ¶

Return node names in declaration order.

`edges_from(node_name)` ¶

Return edges whose producer is node_name.

`edges_to(node_name)` ¶

Return edges whose consumer is node_name.

Local runtime and reports¶

`etlantic.runtime` ¶

Local runtime package.

`CancellationPolicy` `dataclass` ¶

Cancellation behavior.

`InvalidationMode` ¶

Bases: StrEnum

How prior artifacts are invalidated for a rerun.

`MaterializationPolicy` ¶

Bases: StrEnum

How intermediate artifacts should be materialized for a run.

`RetryPolicy` `dataclass` ¶

Retry behavior for steps and runs.

`RunIntent` ¶

Bases: StrEnum

Why a pipeline run was requested.

`RunRequest` `dataclass` ¶

Portable request describing how to execute a pipeline.

`asset_overrides` `property` ¶

Preferred public view of node → logical asset overrides.

`RunSelection` `dataclass` ¶

Graph selection describing which nodes participate in a run.

`resolve(graph)` ¶

Resolve this selection to declaration-ordered node names.

`to_plan_selection(graph)` ¶

Convert to planner selection dict.

`TimeoutPolicy` `dataclass` ¶

Timeout behavior for runs and steps.

`FailureStage` ¶

Bases: StrEnum

Where a node failure occurred.

`RunStatus` ¶

Bases: StrEnum

Normalized pipeline run status.

`StepStatus` ¶

Bases: StrEnum

Normalized step / node execution status.

`etlantic.lifecycle` ¶

Lifecycle extension package.

`CallbackRegistry` `dataclass` ¶

Lifecycle outcome callbacks.

`FailureAction` ¶

Bases: StrEnum

How the runtime should proceed after a callback.

`StepFailureContext` `dataclass` ¶

Context passed to step-failure callbacks.

`MiddlewareStack` `dataclass` ¶

Deterministic ordered middleware stack.

`Emit` `dataclass` ¶

Bases: Generic[T]

Runtime emission of an outbound event (payload must be secret-free).

`OutboundEvent` `dataclass` ¶

Bases: Generic[T]

Declared outbound event that a pipeline may emit.

`Inject` `dataclass` ¶

Annotation marker for hierarchical resource injection.

`ResourceManager` `dataclass` ¶

Scoped resource acquisition with yield cleanup exactly once.

`PipelineRuntime` `dataclass` ¶

Process-scoped runtime coordinating local execution.

Owns in-memory and file-backed storage bindings, plugin registries, middleware stacks, secret providers, and run reports. Application code creates one runtime per process (or per test) and passes it to :meth:~etlantic.pipeline.Pipeline.run.

Call :meth:ensure_plugins_for_profile before the first run when using optional engine plugins so entry points are authorized and loaded for the active :class:~etlantic.profile.Profile.

Built-in storage ids include memory, local, callable, json, csv, and null. Seed in-memory assets with :attr:memory (:class:~etlantic.storage.memory.MemoryStorage).

`ensure_plugins_for_profile(profile)` ¶

Discover and load plugins authorized for profile (0.20).

Idempotent per profile key. No entry points are imported until this method runs (or manual register_*_plugin calls).

`add_run_middleware(middleware, *, name=None)` ¶

Register middleware invoked around entire pipeline runs.

Parameters:

Name	Type	Description	Default
`middleware`	`Any`	Callable or async callable conforming to the run middleware protocol.	required
`name`	`str \| None`	Optional stable name for ordering and diagnostics.	`None`

`add_step_middleware(middleware, *, name=None)` ¶

Register middleware invoked around individual step execution.

Parameters:

Name	Type	Description	Default
`middleware`	`Any`	Callable or async callable conforming to the step middleware protocol.	required
`name`	`str \| None`	Optional stable name for ordering and diagnostics.	`None`

`override_resource(name, provider)` ¶

Replace a named injectable resource provider for this runtime.

Parameters:

Name	Type	Description	Default
`name`	`str`	Resource key referenced by :class:`~etlantic.lifecycle.Inject`.	required
`provider`	`Callable[..., Any]`	Factory callable resolved at execution time.	required

`register_secret_provider(name, provider)` ¶

Register a secret provider under name.

Parameters:

Name	Type	Description	Default
`name`	`str`	Provider id referenced by profile `secret_providers`.	required
`provider`	`SecretProvider`	Implementation of :class:`~etlantic.secrets.provider.SecretProvider`.	required

`register_storage(name, binding)` ¶

Register a storage binding under name.

Parameters:

Name	Type	Description	Default
`name`	`str`	Provider id referenced by profile `assets` / bindings.	required
`binding`	`StorageBinding`	Storage implementation (JSON, CSV, SQL-backed, …).	required

`register_dataframe_plugin(engine, plugin)` ¶

Register a live dataframe plugin and its planning descriptor.

`register_sql_plugin(engine, plugin)` ¶

Register a live SQL plugin and its planning descriptor.

`register_spark_plugin(engine, plugin)` ¶

Register a live Spark plugin and its planning descriptor.

`register_spark_provider(name, provider)` ¶

Register a live Spark session provider.

`register_orchestrator_plugin(engine, plugin)` ¶

Register a live orchestrator plugin and its planning descriptor.

`register_scheduler_plugin(name, plugin)` ¶

Register a live ExecutionScheduler plugin and its planning descriptor.

`apply_plugin_allowlist(profile)` ¶

Filter discovered plugins using profile.plugin_allowlist.

Deprecated: prefer :meth:ensure_plugins_for_profile which authorizes before import. This method re-runs profile-aware discovery.

`session()` `async` ¶

Enter runtime lifespan (if any).

`etlantic.reports` ¶

Run report package.

`FileReportStore` `dataclass` ¶

Durable report store writing one JSON file per run_id via SafeIoPolicy.

`ArtifactResult` `dataclass` ¶

Artifact produced or reused during a run.

`BackendRunReference` `dataclass` ¶

Reference to a backend-specific run identity.

`PipelineRunReport` `dataclass` ¶

Canonical secret-free summary of a pipeline run.

`from_dict(data)` `classmethod` ¶

Deserialize a run report from its JSON-friendly dict form.

Requires schema equal to :data:REPORT_SCHEMA. Missing or unknown schemas are rejected (no silent default). Documents are upgraded via :func:etlantic.reports.upgrade.upgrade_report_dict first.

`RunDiagnostic` `dataclass` ¶

Execution diagnostic (secret-free).

`RunRecommendation` `dataclass` ¶

Suggested follow-up action after a run.

`RunSummary` `dataclass` ¶

Aggregate counters for a run.

`SchemaObservationResult` `dataclass` ¶

Schema observation recorded during a run.

`StateTransitionResult` `dataclass` ¶

Recorded state transition.

`StepRunReport` `dataclass` ¶

Per-step execution summary (secret-free).

`ValidationResult` `dataclass` ¶

Runtime validation outcome at a boundary.

`ReportStore` `dataclass` ¶

Process-local store of completed/partial run reports.

`compare_reports(left, right)` ¶

Compare two normalized reports without backend-specific classes.

`render_html(report)` ¶

Render a minimal HTML report (secret-free).

`render_text(report)` ¶

Render a compact human-readable report.

Storage and secrets¶

`etlantic.storage` ¶

Local storage bindings for ETLantic 0.4.

`CallableStorage` ¶

User-registered Python readers/writers keyed by binding name.

`CsvStorage` ¶

Read/write CSV files using ContractModel field order when available.

`JsonStorage` ¶

Read/write JSON arrays or JSON Lines files.

`MemoryStorage` ¶

Named in-process datasets keyed by binding or location.

`seed(binding, data, *, location=None)` ¶

Seed data for tests and callable pipelines.

`NullStorage` ¶

Read empty datasets; discard writes (VALIDATE / no-write intents).

`StorageBinding` ¶

Bases: Protocol

Read/write datasets for Extract and Load nodes.

`as_records(data, contract_type)` ¶

Normalize data to a list of contract instances or mappings.

`records_to_dicts(data)` ¶

Convert records to plain dicts for file writers.

`etlantic.secrets` ¶

Runtime secret resolution package.

`SecretCache` `dataclass` ¶

Process-local bounded secret cache (never serialized).

`revoke(*, provider=None, name=None)` ¶

Invalidate matching entries; return count removed.

`EnvSecretProvider` ¶

Resolve secrets from process environment variables.

Looks up {name} or {name}_{key} when key is not the default. Fail-closed: missing values raise; no silent empty fallback.

`MountedFileSecretProvider` ¶

Resolve secrets from files under a mount root.

Default path: {root}/{name} or {root}/{name}/{key}. Fail-closed on missing/unreadable files.

`ProviderContext` `dataclass` ¶

Context for provider lifespan.

`SecretProvider` ¶

Bases: Protocol

Protocol for runtime secret resolution.

`SecretProviderCapabilities` `dataclass` ¶

Declared secret-provider capabilities.

`SecretProviderDescriptor` `dataclass` ¶

Installed secret provider metadata.

`SecretResolutionContext` `dataclass` ¶

Caller identity for a secret resolution (no values).

`SecretRef` `dataclass` ¶

Logical reference to a secret; never contains the secret value.

`identity()` ¶

Deterministic identity for this reference.

`to_dict()` ¶

Serialize for plans and profiles (secret-free).

`from_dict(data)` `classmethod` ¶

Deserialize a SecretRef from a mapping.

`SecretSerializationError` ¶

Bases: TypeError

Raised when a SecretValue is asked to serialize.

`SecretValue` `dataclass` ¶

Runtime-only secret payload with redacted display.

Must never appear in plans, reports, events, or logs. Prefer passing this only to the declared resource consumer.

`value` `property` ¶

Return the underlying secret payload.

`get_secret_value()` ¶

Alias for frameworks that expect pydantic-style accessors.

`to_dict()` ¶

Refuse serialization of secret values.

Contract interchange¶

ODCS / DTCS / DPCS loading, diffs, and bundle helpers:

`etlantic.interchange` ¶

Contract interchange: ODCS, DTCS, DPCS, bundles, and diffs.

`ContractBundle` `dataclass` ¶

In-memory view of a generated or loaded contract bundle.

`ArtifactProvenance` `dataclass` ¶

Origin metadata for a loaded or generated artifact.

`ProvenanceKind` ¶

Bases: StrEnum

How a logical artifact was obtained.

`generate_contracts(pipeline_cls)` ¶

Discover contracts and build an in-memory bundle for pipeline_cls.

`load_bundle(directory)` ¶

Load a contract bundle directory and reconstruct the pipeline class.

`write_contracts(pipeline_cls, directory)` ¶

Generate and write a deterministic contract bundle under directory.

`diff_data_contracts(previous, current, *, mode=CompatibilityMode.BACKWARD)` ¶

Compare two data contracts via ContractModel and return diagnostics.

`diff_pipelines(previous, current)` ¶

Compare two pipelines / DPCS docs via the dpcs toolkit.

`diff_transformations(previous, current)` ¶

Compare two transformations / DTCS docs via the dtcs toolkit.

`pipeline_from_dpcs(source, *, registry=None, root=None, class_name=None)` ¶

Load a DPCS artifact into a dynamic Pipeline subclass.

`pipeline_to_dpcs(pipeline_cls, *, data_locations=None, transform_locations=None)` ¶

Build a DPCS document dict from a Pipeline subclass.

`write_dpcs(pipeline_cls, path, **kwargs)` ¶

Write a Pipeline to a DPCS YAML file.

`transformation_from_dtcs(source, *, contracts=None, root=None, class_name=None)` ¶

Load a DTCS artifact into a dynamic Transformation subclass.

`transformation_to_dtcs(transformation_cls)` ¶

Build a DTCS document dict from a Transformation subclass.

`write_dtcs(transformation_cls, path)` ¶

Write a Transformation to a DTCS YAML file.

`graphs_equivalent(left, right)` ¶

Return True when two logical graphs match by fingerprint.

`normalize_pipeline(pipeline_cls)` ¶

Return the logical graph and provenance for a pipeline class.

`load_data_contract(path, *, root=None, class_name=None)` ¶

Load an ODCS file into a ContractModel subclass via ContractModel.

`write_odcs(model, path, *, root=None)` ¶

Write a ContractModel class to an ODCS YAML file.

Gate A tabular interchange (`etlantic.interchange/1`)¶

Available in ETLantic 0.21.0. Versioned, capability-driven tabular interchange for Polars ↔ Pandas boundaries. PySpark/SQL Gate A pairs are not in scope yet. Legacy Arrow-assisted helpers (when PyArrow is installed) are not the Gate A contract.

Planner and runtime use descriptors, mechanism selection, fidelity checks, and evidence types from etlantic.interchange.tabular. Adopter guides: Interchange Gate A FAQ, Polars ↔ Pandas example.

`etlantic.interchange.tabular` ¶

Versioned, capability-driven tabular interchange contracts.

`InterchangeBounds` `dataclass` ¶

Initial Gate A batching and buffering limits.

`CopyEligibility` ¶

Bases: StrEnum

Planned copy behavior for an interchange boundary.

`InterchangeDescriptor` `dataclass` ¶

Secret-free physical interchange decision recorded in a plan.

`to_dict()` ¶

Serialize the descriptor using only JSON-compatible values.

`from_dict(data)` `classmethod` ¶

Validate and deserialize an interchange descriptor.

`fingerprint_inputs()` ¶

Return normalized decision inputs suitable for stable hashing.

`InterchangeDescriptorError` ¶

Bases: InterchangeError

Raised when an interchange descriptor fails closed validation.

`InterchangeError` ¶

Bases: ValueError

Base error for tabular interchange.

`InterchangeSelectionError` ¶

Bases: InterchangeError

Raised when no contract-safe interchange mechanism can be selected.

`InterchangeEvidence` `dataclass` ¶

Runtime observations used to substantiate interchange claims.

`can_report_zero_copy(eligibility)` ¶

Return whether planned eligibility and observations prove zero copy.

`FidelityResult` `dataclass` ¶

Aggregate result of mapping fidelity evaluation.

`accepted` `property` ¶

Return whether the mapping preserves required semantics.

`FidelityStatus` ¶

Bases: StrEnum

Outcome of evaluating physical mapping issues.

`MappingIssue` `dataclass` ¶

A type-family mapping concern discovered before interchange.

`InterchangeMechanism` ¶

Bases: StrEnum

Supported physical mechanisms for tabular interchange.

`check_mapping_fidelity(issues)` ¶

Evaluate mapping issues without side effects.

`evaluate_fidelity(issues)` ¶

Return fail with reasons when any mapping issue is lossy.

`select_mechanism(producer_caps, consumer_caps, *, durable, already_collecting, pyarrow_available, mapping_lossy=False)` ¶

Select the Gate A mechanism from declared capabilities.

Lossy logical mappings fail before a fallback or physical mutation can occur. All fallback selections carry an explicit reason.

`validate_descriptor(data)` ¶

Validate an exact etlantic.interchange/1 descriptor.

API — Plan and Runtime¶

0.21 trust and plan essentials¶

Validation and diagnostics¶

etlantic.diagnostics ¶

Severity ¶

SourceLocation dataclass ¶

DiagnosticAction dataclass ¶

to_dict() ¶

Diagnostic dataclass ¶

ValidationReport dataclass ¶

valid property ¶

has_errors property ¶

errors property ¶

warnings property ¶

filter(*, severity=None, code=None, phase=None) ¶

raise_for_errors() ¶

merge(other) ¶

from_diagnostics(diagnostics, *, phases=()) classmethod ¶

codes() ¶

etlantic.validation ¶

validate_pipeline(pipeline_cls, *, context=None, profile=None, policy=None) ¶

validate_transformation(transform) ¶

etlantic.policy ¶

PolicyMode ¶

ValidationPolicy dataclass ¶

to_dict() ¶

register_validation_policy(policy) ¶

resolve_validation_policy(name) ¶

Profiles, planning, and registries¶

etlantic.profile ¶

Profile dataclass ¶

engine_profile property ¶

assets property ¶

primary_engine() ¶

identity() ¶

to_dict() ¶

to_plan_snapshot() ¶

from_plan_snapshot(data) classmethod ¶

from_dict(data, *, accept_legacy_bindings=False) classmethod ¶

with_updates(**kwargs) ¶

development_profile(**overrides) ¶

test_profile(**overrides) ¶

production_profile(**overrides) ¶

resolve_profile(profile, *, allow_adhoc_profile=False) ¶

write_profile(profile, path) ¶

load_profile(path, *, accept_legacy_bindings=False) ¶

etlantic.plan ¶

ArtifactRef dataclass ¶

to_dict() ¶

from_dict(data) classmethod ¶

ArtifactStrategy ¶

PipelinePlan dataclass ¶

to_dict() ¶

compile(*, target='airflow', profile=None, plugin=None, **kwargs) ¶

from_dict(data, *, verify=True) classmethod ¶

explain_plan(plan) ¶

deep_freeze(value) ¶

validate_plan_interchange(plan) ¶

plan_pipeline(pipeline_cls, *, context=None, profile=None, selection=None) ¶

plan_pipeline_with_report(pipeline_cls, *, context=None, profile=None, selection=None) ¶

canonical_plan_json(plan) ¶

plan_fingerprint(plan) ¶

plan_from_json(text, *, verify=True) ¶

plan_to_json(plan, *, indent=2) ¶

verify_plan_fingerprint(plan) ¶

dependency_closure(graph, targets) ¶

run_one_selection(graph, step_name) ¶

run_until_selection(graph, step_name) ¶

etlantic.registry ¶

PluginDescriptor dataclass ¶

to_dict() ¶

ImplementationDescriptor dataclass ¶

to_dict() ¶

from_dict(data) classmethod ¶

BindingDescriptor dataclass ¶

to_dict() ¶

from_dict(data) classmethod ¶

RegistryBundle dataclass ¶

register_plugin(descriptor) ¶

register_binding(descriptor) ¶

`etlantic.diagnostics` ¶

`Severity` ¶

`SourceLocation` `dataclass` ¶

`DiagnosticAction` `dataclass` ¶

`to_dict()` ¶

`Diagnostic` `dataclass` ¶

`ValidationReport` `dataclass` ¶

`valid` `property` ¶

`has_errors` `property` ¶

`errors` `property` ¶

`warnings` `property` ¶

`filter(*, severity=None, code=None, phase=None)` ¶

`raise_for_errors()` ¶

`merge(other)` ¶

`from_diagnostics(diagnostics, *, phases=())` `classmethod` ¶

`codes()` ¶

`etlantic.validation` ¶

`validate_pipeline(pipeline_cls, *, context=None, profile=None, policy=None)` ¶

`validate_transformation(transform)` ¶

`etlantic.policy` ¶

`PolicyMode` ¶

`ValidationPolicy` `dataclass` ¶

`to_dict()` ¶

`register_validation_policy(policy)` ¶

`resolve_validation_policy(name)` ¶

`etlantic.profile` ¶

`Profile` `dataclass` ¶

`engine_profile` `property` ¶

`assets` `property` ¶

`primary_engine()` ¶

`identity()` ¶

`to_dict()` ¶

`to_plan_snapshot()` ¶

`from_plan_snapshot(data)` `classmethod` ¶

`from_dict(data, *, accept_legacy_bindings=False)` `classmethod` ¶

`with_updates(**kwargs)` ¶

`development_profile(**overrides)` ¶

`test_profile(**overrides)` ¶

`production_profile(**overrides)` ¶

`resolve_profile(profile, *, allow_adhoc_profile=False)` ¶

`write_profile(profile, path)` ¶

`load_profile(path, *, accept_legacy_bindings=False)` ¶

`etlantic.plan` ¶

`ArtifactRef` `dataclass` ¶

`to_dict()` ¶

`from_dict(data)` `classmethod` ¶

`ArtifactStrategy` ¶

`PipelinePlan` `dataclass` ¶

`to_dict()` ¶

`compile(*, target='airflow', profile=None, plugin=None, **kwargs)` ¶

`from_dict(data, *, verify=True)` `classmethod` ¶

`explain_plan(plan)` ¶

`deep_freeze(value)` ¶

`validate_plan_interchange(plan)` ¶

`plan_pipeline(pipeline_cls, *, context=None, profile=None, selection=None)` ¶

`plan_pipeline_with_report(pipeline_cls, *, context=None, profile=None, selection=None)` ¶

`canonical_plan_json(plan)` ¶

`plan_fingerprint(plan)` ¶

`plan_from_json(text, *, verify=True)` ¶

`plan_to_json(plan, *, indent=2)` ¶

`verify_plan_fingerprint(plan)` ¶

`dependency_closure(graph, targets)` ¶

`run_one_selection(graph, step_name)` ¶

`run_until_selection(graph, step_name)` ¶

`etlantic.registry` ¶

`PluginDescriptor` `dataclass` ¶

`to_dict()` ¶

`ImplementationDescriptor` `dataclass` ¶

`to_dict()` ¶

`from_dict(data)` `classmethod` ¶

`BindingDescriptor` `dataclass` ¶

`to_dict()` ¶

`from_dict(data)` `classmethod` ¶

`RegistryBundle` `dataclass` ¶

`register_plugin(descriptor)` ¶

`register_binding(descriptor)` ¶

`register_implementation(descriptor)` ¶

`PlanningContext` `dataclass` ¶

`create(profile=None, *, registry=None, required_capabilities=None, allow_capability_fallback=False, allow_adhoc_profile=False)` `classmethod` ¶

`builtin_stub_registry()` ¶