API — Optional packages¶
Status: Available in ETLantic 0.41.0. Per-package API pages for first-party optional packages. Install/overview hub: Optional Packages. Core symbols: Python API Reference.
Each package page includes a minimal setup example, a failure-mode table, and mkdocstrings coverage of the public module tree. Package READMEs remain the install and narrative home.
Packages¶
- etlantic-polars — Polars dataframe plugin + portable compiler
- etlantic-pandas — Pandas dataframe plugin + portable compiler
- etlantic-sql — SQL plugin (SQLite + PostgreSQL)
- etlantic-pyspark — PySpark plugin + portable compiler
- etlantic-airflow — Airflow DAG compiler
- etlantic-prefect — Prefect local scheduler MVP
- etlantic-keyring — OS keyring secret provider
- etlantic-sqlmodel — SQLModel bridge helpers
- etlantic-datafusion — Experimental DataFusion stub
- etlantic-s3 — Experimental S3-compatible connectors
- etlantic-iceberg — Experimental Iceberg connectors
- etlantic-snowflake — Experimental Snowflake connectors
- etlantic-openlineage — Experimental outbound OpenLineage (CP2)
- etlantic-fastapi — FastAPI reference adapter
- medallantic — Medallion facade + SparkForge migrate
Quick module stubs (roots)¶
The directives below keep root-level coverage discoverable from this hub:
etlantic-polars¶
etlantic-pandas¶
etlantic-sql¶
etlantic-sql — PostgreSQL / SQLite Tier A reference SQL execution plugin.
etlantic-pyspark¶
etlantic-airflow¶
etlantic-airflow — Airflow reference orchestrator for ETLantic 0.8.
Compile a PipelinePlan into a deterministic Airflow DAG module.
Import and return the generated module (exposes dag and plan meta).
Import a generated DAG module and return its dag object.
Requires apache-airflow installed in the Airflow environment.
Entry-point factory for etlantic.orchestrator_plugins.
etlantic-prefect¶
etlantic-keyring¶
etlantic-keyring — OS keyring secret provider for ETLantic.
Resolve secrets from the OS keyring via keyring.get_password.
SecretRef.name is the keyring service; SecretRef.key is the username.
When key is value, default, or empty, name is used as the
username and the configured default service applies.
Factory for runtime.register_secret_provider(...) / profile bindings.
etlantic-sqlmodel¶
etlantic-sqlmodel — ContractModel ↔ SQLModel bridge + CP1 reference stores.
Bases: SQLModel
Scoped alias pointing at an immutable revision.
Bases: SQLModel
Deployment / promotion environment directory row.
Bases: SQLModel
Stable logical identity preserved across revisions.
Bases: SQLModel
Immutable promotion record preserving logical_id.
Bases: SQLModel
Immutable append-only registry revision.
Bases: SQLModel
Security-domain directory row.
Workspace-scoped definition registry backed by SQLModel.
Minimal SQLModel-backed EventStore with tenant/workspace isolation.
dataclass
¶
SqlModelRegistryProvider(engine: Engine, tenants: SqlModelTenantDirectory | None = None, workspaces: SqlModelWorkspaceDirectory | None = None, revisions: SqlModelRevisionRegistry | None = None)
SQLModel :class:RegistryProvider with shared suspension checks.
dataclass
¶
SqlModelRevisionRegistry(engine: Engine, tenants: SqlModelTenantDirectory | None = None, workspaces: SqlModelWorkspaceDirectory | None = None)
Append-only SQLModel revision store with aliases and promotions.
Durable acceptance store with ADR-016 scoped idempotency and run observation.
dataclass
¶
SQLModel tenant directory with isolation and suspension fail-closed.
dataclass
¶
SQLModel workspace directory keyed by (tenant_id, workspace_id).
Bases: SQLModel
Durable tenant directory row (CP2 / 040-P).
Bases: SQLModel
Durable workspace directory row (tenant-owned).
Schema bridge helpers without ORM sessions or migrations.
Create CP reference tables.
Intended for tests and local demos — not a production migration path.
Create CP2 registry tables.
Intended for tests and local demos — not a production migration path.
Prefer :mod:etlantic_sqlmodel.migrations.
Create an engine; use a file URL for restart durability tests.
Return a zero-arg factory that opens a short-lived Session.
FastAPI-compatible dependency generator for a request-scoped session.
Sessions stay in the API layer — never pass them into pipeline runtimes.
Yield a session that commits on success and rolls back on error.
Upgrade to the latest migration (SQLite-friendly for tests).
Return the applied migration version, or None on a fresh database.
Roll back migrations down to target (None = empty schema).
Apply pending migrations up to target (default: latest).
contract_to_sqlmodel(contract_cls: type[Data], *, table_name: str | None = None, primary_key: tuple[str, ...] | None = None) -> type[SQLModel]
Generate a SQLModel table class from a Data / ContractModel class.
contract_to_sqlmodel_source(contract_cls: type[Data], *, table_name: str | None = None, primary_key: tuple[str, ...] | None = None) -> str
Generate importable SQLModel source for a Data / ContractModel class.
Extract draft contract metadata from a SQLModel table class.
Return ordered primary-key column names from a SQLModel table class.
validate_model_primary_keys(model_cls: type[Any], *, expected_keys: tuple[str, ...] | list[str] | None = None) -> ValidationReport
Fail closed when a SQLModel table lacks the expected primary key.
Compare contract and SQLModel metadata; return a ValidationReport.
Entry-point factory for tooling and conformance helpers.
run_conformance_checks(contract_cls: type[Data], *, table_name: str | None = None, primary_key: tuple[str, ...] | None = None) -> ValidationReport
Round-trip contract → SQLModel → metadata and compare schemas.
etlantic-datafusion¶
Experimental DataFusion plugin package (stub; not production-ready in 0.38.0).
etlantic-s3¶
Experimental S3-compatible connector package for ETLantic.
dataclass
¶
S3SinkConnector(backend: InMemoryS3Fake = InMemoryS3Fake(), force_fake: bool = True, _sessions: dict[str, dict[str, Any]] = dict())
Stage multipart uploads; commit via conditional pointer publication.
dataclass
¶
Read committed S3 objects via commit-pointer resolution.
dataclass
¶
Bounded schema inspection over committed S3 objects.
dataclass
¶
InMemoryS3Fake(objects: dict[tuple[str, str], bytes] = dict(), etags: dict[tuple[str, str], str] = dict(), multiparts: dict[str, _MultipartUpload] = dict(), commit_pointers: dict[tuple[str, str], str] = dict(), _ops: list[dict[str, Any]] = list())
Deterministic S3 stub for unit tests (no network).
Semantics mirrored for CI:
- Multipart uploads stage parts until complete or abort.
- Abort discards parts and marks the upload aborted (orphan cleanup).
- Commit pointers use conditional put (If-None-Match
*by default): only one winner; losers see a precondition failure for reconciliation. - Readers resolve only the committed pointer, never incomplete multipart staging keys.
put_commit_pointer(*, bucket: str, pointer_key: str, data_key: str, if_none_match: bool = True) -> dict[str, Any]
Conditionally publish a commit pointer to an immutable data object.
When if_none_match is True (default), succeeds only if the pointer
does not already exist — concurrent losers get precondition_failed.
Resolve pointer → immutable object; ignore incomplete multipart keys.
Entry-point factory for etlantic.source_connectors.
Entry-point factory for etlantic.storage_connectors.
Return True when boto3 can be imported (live path opt-in).
etlantic-iceberg¶
etlantic-snowflake¶
Experimental Snowflake connector package for ETLantic.
dataclass
¶
FakeSnowflakeConnection(autocommit: bool = False, tables: dict[str, list[dict[str, Any]]] = dict(), _txn: list[tuple[str, str, list[dict[str, Any]]]] = list(), _committed_queries: dict[str, FakeQueryResult] = dict(), _pending_queries: dict[str, FakeQueryResult] = dict(), _query_seq: int = 0)
Transactional fake: autocommit is always off.
Staged DML lives in a transaction buffer until commit() or rollback().
Every statement receives a stable query_id for CommitReceipt evidence and
post-loss reconciliation.
etlantic-fastapi¶
Thin FastAPI adapter: control-plane API (CP1) + non-CP reference app.
Package version is 0.39.0 (CP1 gate-ready).
dataclass
¶
AuthoringService(policy: PolicyContext = PolicyContext(), runtime: PipelineRuntime = PipelineRuntime(), definitions: dict[str, PipelineDefinition] = dict(), jobs: dict[str, RunJob] = dict())
In-memory facade mapping request models to public ETLantic operations.
Not a multi-tenant control plane. HTTP adapters (for example
etlantic-fastapi) should bind host policy into PolicyContext.
Return capability negotiation metadata for clients.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Capability negotiation payload plus |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
Return the authoring catalog, optionally scoped to a definition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str | None
|
Optional stored definition id to enrich the catalog. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Catalog document as a dict. |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
put_definition(definition_id: str, document: dict[str, Any], *, idempotency_key: str | None = None) -> dict[str, Any]
Store a verified pipeline definition document.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Host-assigned definition id. |
required |
document
|
dict[str, Any]
|
|
required |
idempotency_key
|
str | None
|
Reserved for adapters; ignored in this reference. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Mapping with |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
ValueError
|
On fingerprint / schema verification failure. |
TypeError
|
If |
Fetch a stored definition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Previously stored id. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Mapping with |
Raises:
| Type | Description |
|---|---|
KeyError
|
If |
apply_edit(definition_id: str, command: dict[str, Any], *, expected_token: str | None = None) -> dict[str, Any]
Apply an immutable edit to a stored definition.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Stored definition id. |
required |
command
|
dict[str, Any]
|
|
required |
expected_token
|
str | None
|
Optimistic concurrency token (fingerprint). |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Updated id, fingerprint, concurrency token, and definition. |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the definition id is unknown. |
ValueError
|
On concurrency or edit payload failures. |
Structurally validate a stored definition for the policy profile.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Stored definition id. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Mapping with |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the definition id is unknown. |
Plan a stored definition for the policy profile.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Stored definition id. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Mapping with |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the definition id is unknown. |
Run synchronously (reference adapter — not a durable async queue).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
definition_id
|
str
|
Stored definition id. |
required |
idempotency_key
|
str | None
|
Reserved for adapters; ignored here. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Job status mapping from |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the definition id is unknown. |
Reference adapter runs are synchronous; cancel is not supported in-flight.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_id
|
str
|
Job id from |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Status mapping with |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the job id is unknown. |
Return status for a synchronous reference job.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
job_id
|
str
|
Job id from |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Status mapping including optional |
Raises:
| Type | Description |
|---|---|
PermissionError
|
If |
KeyError
|
If the job id is unknown. |
dataclass
¶
PolicyContext(tenant: str = 'default', environment: str = 'development', profile: str = 'development', allowed_assets: tuple[str, ...] | None = None, allowed_plugins: tuple[str, ...] | None = None, allowed_actions: tuple[str, ...] = ('catalog', 'validate', 'plan', 'edit', 'run', 'cancel', 'report'))
Host-supplied authority; client payloads cannot expand these grants.
Attributes:
| Name | Type | Description |
|---|---|---|
tenant |
str
|
Logical tenant label for the host (reference only). |
environment |
str
|
Deployment environment label. |
profile |
str
|
Profile name resolved for validate/plan/run. |
allowed_assets |
tuple[str, ...] | None
|
When set, definition assets must be in this set. |
allowed_plugins |
tuple[str, ...] | None
|
When set, profile allowlist and engine refs must match. |
allowed_actions |
tuple[str, ...]
|
Actions the facade may perform. |
dataclass
¶
ETLanticAPI(authorizer: Authorizer, definitions: DefinitionRepository, submissions: SubmissionStore, events: EventStore, context_factory: ContextFactory, principal_dependency: PrincipalDependency = principal_from_header, profile: Any = 'development', history_store: HistoryStore | None = None, known_observation_ids: set[str] = set(), registry: RegistryProvider | None = None, durable_work: DurableWorkStore | None = None, title: str = 'ETLantic Control Plane', version: str = __version__)
Control-plane API holding injected stores and auth adapters.
Heavy pipeline work is never scheduled via FastAPI BackgroundTasks. Submission acceptance is a durable store commit only; optional pollers may observe accepted jobs without executing them in-request.
Optional registry enables /v1/registry admin routes (CP2). Use
:meth:with_registry_definitions or definitions_backend="registry" so
/v1/definitions* read/write through registry revisions without changing
route paths or operationIds.
Optional durable_work enables /v1/durable/* host routes (CP3) and
dual-writes durable accept on submit when fingerprints are present.
classmethod
¶
with_registry_definitions(*, authorizer: Authorizer, registry: RegistryProvider, submissions: SubmissionStore, events: EventStore, context_factory: ContextFactory, principal_dependency: PrincipalDependency | None = None, **kwargs: Any) -> ETLanticAPI
Build an API whose /v1/definitions* use registry-backed storage.
dataclass
¶
LandingWatchSubmitter(watch_dir: Path, definition_id: str, submit_run: SubmitRun, binding_ref: Mapping[str, Any] = local_files_binding_ref(), tenant_id: str = 'default', workspace_id: str = 'default', poll_interval: float = 0.25, pattern: str = '*.csv', _seen: MutableMapping[str, float] = dict())
Stdlib polling loop that submits durable runs for new files.
Prefer this over optional watchdog so CP1 has zero extra deps.
create_app(api: ETLanticAPI | None = None, *, authorizer: Authorizer | None = None, definitions: DefinitionRepository | None = None, submissions: SubmissionStore | None = None, events: EventStore | None = None, context_factory: ContextFactory | None = None, principal_dependency: PrincipalDependency | None = None, registry: RegistryProvider | None = None, durable_work: DurableWorkStore | None = None, definitions_backend: str | None = None, title: str | None = None, version: str | None = None, install_handlers: bool = True, with_lifespan: bool = True) -> FastAPI
Standalone control-plane app factory.
Optional lifespan wires injected stores onto app.state and verifies
readiness. It does not start BackgroundTasks or execute pipelines.
definitions_backend="registry" requires registry and wraps it as
:class:RegistryDefinitionRepository for stable /v1/definitions*
routes. Default keeps an injected definitions store (callers must pass
one when not using the registry backend).
Embed the control-plane router without owning host lifecycle.
Does not install lifespan hooks, middleware, or exception handlers. Host applications should register Problem Details handlers and lifespan themselves when desired.
Build a principal dependency that reads header from the request.
Build a context factory that maps principal subjects via membership.
oauth2_oidc_principal_hook(*, token_claims: Mapping[str, Any], subject_claim: str = 'sub', issuer_claim: str = 'iss') -> Principal
Placeholder OAuth2/OIDC claim → Principal mapping for host adapters.
Hosts that validate JWTs (or introspect tokens) via FastAPI security dependencies can call this after claims are verified. This function does not validate signatures or contact an IdP.
Wrap a host-defined principal resolver as a FastAPI dependency.
Demo/test principal adapter reading an opaque X-Principal header.
Production hosts should replace this with OAuth2/OIDC or mTLS mapping.
static_context_factory(*, tenant_id: str, workspace_id: str, environment: str = 'development', security_domain: str = 'default') -> ContextFactory
Fixed-scope context factory for single-tenant demos and unit tests.
assert_path_scope(ctx: ControlPlaneContext, *, tenant_id: str | None = None, workspace_id: str | None = None) -> None
Reject path scope that does not match server-derived context (404).
Spoofed path tenants must not disclose existence — map to opaque not_found.
Build a dependency that constructs server-derived context.
Path tenant/workspace segments are never treated as authority. When a
route exposes those path params for routing, callers should verify them
against the returned context via :func:assert_path_scope.
Map :class:ControlPlaneError to an RFC 7807-shaped JSON body.
Install CP error handlers on a standalone app (not used by include_router).
local_files_binding_ref(*, root_ref: str = 'landing', root: str = 'inbox', glob: str = '*.csv', mode: str = 'snapshot', provider: str = 'local-files', format: str = 'csv') -> dict[str, Any]
0.38 local-files-style binding reference (paths/refs only, no bytes).
Adapt a FastAPI TestClient into a :data:SubmitRun callable.
create_reference_app(*, service: AuthoringService | None = None, title: str = 'ETLantic Authoring Reference', version: str = __version__) -> FastAPI
Create a FastAPI app exposing the public authoring/service facade.
This is a non-CP sync demo. It is not the control plane and must not be treated as a multi-tenant production surface.
Encode one ControlPlaneEvent as an SSE message (id = resume cursor).
sse_streaming_response(events: EventStore, ctx: ControlPlaneContext, run_id: str, *, cursor: str | None, follow: bool = False, headers: Mapping[str, str] | None = None, max_polls: int | None = None, max_duration_s: float | None = None) -> StreamingResponse
Build a text/event-stream response for run events.
medallantic¶
Medallantic — engine-agnostic medallion pipelines built on ETLantic.
dataclass
¶
AdaptationResult(pipeline_cls: type[Pipeline], profile: Profile, validation_policy: ValidationPolicy, write_intents: tuple[WriteIntent, ...] = (), step_map: dict[str, str] = dict(), layer_by_node: dict[str, str] = dict(), diagnostics: tuple[Diagnostic, ...] = (), metadata: dict[str, Any] = dict(), required_delta_operations: tuple[str, ...] = ())
Result of adapting a SparkForge pipeline IR.
Bases: Exception
Raised when SparkForge → ETLantic adaptation fails closed.
Bronze(*, asset: str | None = None, source: str | None = None, write_mode: str | None = None, transform_ref: str | None = None, rules: dict[str, Any] | None = None, description: str | None = None, tags: tuple[str, ...] | list[str] | None = None, kind: str | None = None, metadata: dict[str, Any] | None = None, name: str | None = None)
Class-style native medallion authoring.
Subclass and declare Bronze / Silver / Gold attributes, then call
to_definition() or lower().
classmethod
¶
Return the facade-owned medallion document for this class.
classmethod
¶
Build an anonymous MedallionPipeline subclass from a document dict.
classmethod
¶
Lower this pipeline onto ETLantic Pipeline / definition surfaces.
classmethod
¶
Return a sealed public PipelineDefinition.
Bases: StrEnum
SparkForge medallion layer (adapter-only vocabulary).
dataclass
¶
SparkForgePipelineSpec(name: str, schema: str = 'default', steps: tuple[SparkForgeStepSpec, ...] = (), min_bronze_rate: float = 90.0, min_silver_rate: float = 95.0, min_gold_rate: float = 98.0, engine: str = 'spark', legacy_engine_extensions: tuple[str, ...] = (), metadata: dict[str, Any] = dict())
Complete SparkForge pipeline description (fixture-friendly).
classmethod
¶
Parse IR and return any coercion / hygiene diagnostics.
dataclass
¶
SparkForgeStepSpec(name: str, kind: StepKind, layer: LayerKind, source: str | None = None, table_name: str | None = None, transform_ref: str | None = None, rules: dict[str, Any] = dict(), write_mode: str | None = None, metadata: dict[str, Any] = dict())
One SparkForge step in secret-free form.
Bases: StrEnum
SparkForge step kinds represented in the compatibility IR.
Bases: Exception
Raised when medallion → ETLantic lowering fails closed.
dataclass
¶
LoweringResult(pipeline_cls: type[Pipeline], profile: Profile, validation_policy: ValidationPolicy, write_intents: tuple[WriteIntent, ...] = (), step_map: dict[str, str] = dict(), layer_by_node: dict[str, str] = dict(), diagnostics: tuple[Diagnostic, ...] = (), metadata: dict[str, Any] = dict(), required_delta_operations: tuple[str, ...] = ())
Bases: Data
Generic row contract for medallion planning/parity (passthrough).
Bases: ValueError
Raised when a Medallantic rules shorthand cannot be parsed.
dataclass
¶
MedallionDocument(name: str, schema: str = 'default', steps: tuple[MedallionStep, ...] = (), min_bronze_rate: float = 90.0, min_silver_rate: float = 95.0, min_gold_rate: float = 98.0, engine: str = 'local', description: str | None = None, tags: tuple[str, ...] = (), metadata: dict[str, Any] = dict())
Declarative medallion pipeline document owned by Medallantic.
dataclass
¶
MedallionStep(name: str, layer: str, kind: str, source: str | None = None, asset: str | None = None, transform_ref: str | None = None, rules: dict[str, Any] = dict(), write_mode: str | None = None, description: str | None = None, tags: tuple[str, ...] = (), metadata: dict[str, Any] = dict())
One bronze, silver, or gold step in a medallion document.
adapt_pipeline(spec: SparkForgePipelineSpec, *, capabilities: PluginCapabilities | None = None, strict_delta: bool = True) -> AdaptationResult
Map a SparkForge pipeline IR to a concrete ETLantic Pipeline subclass.
adapt_profile(spec: SparkForgePipelineSpec, *, name: str | None = None, bindings: dict[str, str] | None = None) -> Profile
Build an ETLantic Profile from SparkForge builder config.
Map layer thresholds onto a named ValidationPolicy (metadata only).
Place serialized write intents under plan.intents['write_intents'].
Create an anonymous MedallionPipeline subclass from a document.
Raises:
| Type | Description |
|---|---|
ValueError
|
When a step declares a layer outside bronze/silver/gold. |
assert_delta_capabilities(operations: list[str], *, capabilities: PluginCapabilities | None = None, strict: bool = True) -> list[Diagnostic]
Fail closed when declared Delta ops are unsupported by capabilities.
When capabilities is None and strict is True (default), emit errors
so callers must supply a plugin that supports spark_delta before
execution. When strict is False (plan-only), emit warnings instead.
Normalize SparkForge retry config into ETLantic RetryPolicy.
Map a SparkForge write-mode string to ETLantic WriteMode.
Extra write-mode metadata preserved when modes collapse (e.g. partitions).
explain_medallion_plan(plan: PipelinePlan, *, definition: PipelineDefinition | None = None) -> dict[str, Any]
Merge core plan explain with medallion layer and lifecycle metadata.
enrich_lifecycle_event(event: LifecycleEvent | SecurityEvent, layer_by_node: dict[str, str]) -> LifecycleEvent | SecurityEvent
Attach medallion layer annotation when step is known.
Summarize run report steps grouped by medallion layer.
lower_document(doc: MedallionDocument, *, required_delta_operations: tuple[str, ...] = (), diagnostic_phase: str = 'medallion_authoring', allowed_transform_refs: tuple[str, ...] | None = None, transform_refs_fail_closed: bool = False, profile: Profile | None = None) -> LoweringResult
Map a medallion document to a concrete ETLantic Pipeline subclass.
Bronze/silver/gold remain facade metadata on layer_by_node; ETLantic
core never sees medallion enums.
Importable transform_ref values are deny-by-default when
transform_refs_fail_closed is True or when profile.security_mode is
production. Pass allowed_transform_refs (or profile metadata
medallantic.allowed_transform_refs) to permit specific callables.
Development profile with medallion observability defaults.
Production template — requires explicit plugin allowlist before deploy.
Test profile with strict validation and in-memory history.
adapt_run_result(payload: dict[str, Any], *, pipeline_id: str | None = None, profile: str = 'medallantic') -> PipelineRunReport
Convert a SparkForge-shaped result dict into PipelineRunReport.
Never retains secret-like keys or free-text credentials from the source
payload. Unknown run statuses fail closed to failed with diagnostic
PMSF500.
enforce_accept_rates(report: PipelineRunReport, *, policy_metadata: dict[str, Any], layer_by_node: dict[str, str] | None = None) -> PipelineRunReport
Fail a successful report when accept-rate thresholds are violated.
Returns the original report when thresholds pass or are absent. On
violation, returns a replaced report with FAILED status and
MDL120 diagnostics.
evaluate_accept_rates(*, policy_metadata: dict[str, Any], validations: list[ValidationResult] | list[dict[str, Any]], layer: str | None = None) -> list[dict[str, Any]]
Compare validation accept rates against Medallantic layer thresholds.
Threshold keys live on ValidationPolicy.metadata as
min_accept_rate_{ingest,clean,publish} (bronze/silver/gold).
Returns a list of findings; empty when thresholds are satisfied or absent.
Dual-reporting helper: ETLantic report → SparkForge-shaped explain dict.
Parse column→shorthand lists into a portable quality ruleset.
Supported shorthands (strings)::
not_null
unique / uniqueness
regex:<pattern>
length:min:max | min_length:N | max_length:N
in:a|b|c
range:min:max
ge:N / le:N / gt:N / lt:N / eq:N / ne:N
custom:<name>
Structured dict forms are also accepted per entry.
Return an ETLantic DebugSession for an adapted pipeline class.
debug_request_from_sparkforge(*, mode: str | None = 'standard', run_until: str | None = None, run_one: str | None = None, run_from: str | None = None, skip_writes: bool = False, materialization: MaterializationPolicy | None = None, retry: dict[str, Any] | RetryPolicy | None = None, parameter_overrides: dict[str, dict[str, Any]] | None = None, implementation_overrides: dict[str, str] | None = None, invalidation: str | InvalidationMode | None = None) -> RunRequest
Build a RunRequest from SparkForge debug/session options.
skip_writes sets no_write=True only. Materialization stays
DEFAULT unless the caller passes materialization= explicitly.
VALIDATE intent also sets no_write=True.
Map SparkForge run-mode strings to RunIntent.
selection_from_sparkforge(*, run_until: str | None = None, run_one: str | None = None, run_from: str | None = None) -> RunSelection
Map SparkForge selective-execution flags to RunSelection.