Skip to content

SparkForge Adapter (runnable)

Status: Available. Uses etlantic-sparkforge and the IR parity suite under tests/sparkforge/.

Map medallion SparkForge IR onto ordinary ETLantic Source / Step / Sink graphs without putting bronze/silver/gold types in core.

Setup

git clone https://github.com/eddiethedean/etlantic.git
cd etlantic
uv sync --group sparkforge

Run the parity suite

uv run pytest -q tests/sparkforge -m sparkforge

Minimal adapt

import json
from pathlib import Path

from etlantic.plan import plan_pipeline
from etlantic_sparkforge import (
    SparkForgePipelineSpec,
    adapt_pipeline,
    debug_request_from_sparkforge,
    enrich_plan,
)

data = json.loads(
    Path("tests/sparkforge/fixtures/ecommerce.json").read_text(encoding="utf-8")
)
spec = SparkForgePipelineSpec.from_dict(data)
adapted = adapt_pipeline(spec)
adapted.pipeline_cls.validate(profile=adapted.profile).raise_for_errors()
plan = enrich_plan(
    plan_pipeline(adapted.pipeline_cls, profile=adapted.profile),
    adapted,
)
request = debug_request_from_sparkforge(mode="incremental", skip_writes=True)
print(adapted.pipeline_cls.__name__, plan.plan_id, request.intent)

Boundary

  • Layer names stay in adapter metadata (layer_by_node), not core enums
  • Transforms may be passthrough for planning parity (PMSF411 warnings)
  • MERGE write intents are for plan/orchestration via enrich_plan; local runtime still gates writes with RunRequest.no_write

See also