Skip to content

Contributing

ETLantic welcomes contributions to documentation, typed authoring APIs, validation, planning, plugins, tests, and examples.

Preserve the boundaries established in the manifesto and foundations documentation: ETLantic owns the logical model; plugins own execution; standards own semantics.

Before You Start

Read:

Scope Test

Ask:

  1. Does this concern portable modeling, validation, or planning?
  2. Is it execution behavior that belongs in a plugin?
  3. Is it data-contract operational behavior that belongs in ContractModel?
  4. Is it contract meaning that belongs in ODCS, DTCS, or DPCS?

ETLantic owns the logical model. Plugins own execution. Standards own semantics.

Development Setup

git clone https://github.com/eddiethedean/etlantic.git
cd etlantic
git checkout -b topic/my-change
uv sync

uv sync installs runtime dependencies, the editable workspace packages, and the dev group (pytest, ruff, mkdocs). Optional groups: dataframes, sql, pyspark, airflow, sparkforge, keyring, sqlmodel. See Installation.

Use the supported Python versions documented in pyproject.toml (3.11+). PySpark real-cluster parity needs a JVM; default Spark tests use sparkless. Airflow tests need the airflow group. PostgreSQL SQL tests need a live URL via ETLANTIC_SQL_URL when not using the SQLite demo path.

CI-equivalent checks

Baseline (core + docs):

uv sync --locked
uv run ruff check .
uv run ruff format --check .
uv run pytest -q -m "not sparkforge and not polars and not pandas and not sql and not spark and not airflow"
uv run python scripts/check_docs.py
uv run python scripts/check_agent_guidance.py
uv run python scripts/check_release.py
uv run python examples/quickstart.py
uv run python scripts/build_docs.py

Optional plugins / portable examples:

uv sync --locked --group dataframes
uv run python examples/portable_polars_kernel.py
uv run python examples/portable_pandas_kernel.py
uv run pytest -q -m "polars or pandas"
uv sync --locked --group sparkforge
uv run pytest -q tests/sparkforge -m sparkforge

Making a Change

  1. Fork and open a branch from main (or identify an issue).
  2. Confirm the architectural owner of the feature.
  3. Add an ADR for difficult-to-reverse architectural changes.
  4. Add tests before or with implementation.
  5. Update affected documentation.
  6. Update CHANGELOG.md under [Unreleased] for user-visible changes (no separate fragment tool is required today).
  7. Run the CI-equivalent checks above.

Pull Requests

Pull requests should include:

  • Problem statement
  • Proposed behavior
  • Public API impact
  • Contract or plugin compatibility impact
  • Tests
  • Documentation changes
  • Performance or security considerations

Keep pull requests focused. Separate unrelated refactoring from behavior changes.

Public API Changes

Changes to root imports, authoring syntax, plugin protocols, PipelinePlan, or generated contract meaning require extra review.

Before adding a public abstraction, demonstrate at least two concrete consumers or one complete end-to-end workflow that needs it.

Documentation Contributions

Documentation should:

  • Lead with the user outcome
  • Use consistent terms from the glossary
  • Distinguish proposed APIs from implemented APIs
  • Link to normative standards instead of duplicating them
  • Include executable examples where possible
  • Avoid claiming that ETLantic executes work owned by plugins

See Documentation Contributions for page-status labels, current-version rules, and CI checks.

Plugin Contributions

Core plugins should:

  • Depend only on public SDK interfaces
  • Declare accurate capabilities
  • Pass conformance tests
  • Normalize diagnostics and failures
  • Avoid importing heavy dependencies until needed
  • Document supported backend versions

Third-party plugins may be maintained independently and distributed through Python package entry points.

Testing

Currently enforced

  • pytest unit/CLI suites on CI
  • ruff check / ruff format --check
  • scripts/check_docs.py + runnable companions + strict MkDocs build
  • Optional dataframe / SparkForge matrix jobs

Currently enforced for portable compilers

  • Public run_portable_transform_conformance_suite for Polars / Pandas / PySpark
  • Hypothesis property tests for capability matching and fingerprint stability
  • Compiler e2e / differential suites in CI dependency-group jobs

Not yet enforced (aspirational)

Coverage thresholds, pyright, dependency audit, and secret scanning are goals—not CI requirements. See Testing and Dependency Strategy.

Run the narrowest relevant tests during development:

uv sync --group dataframes
uv run pytest -m polars
uv run pytest -m pandas
uv run pytest tests/polars_compiler

For Spark plugin work (JVM-free via sparkless by default):

uv sync --group pyspark
uv run pytest tests/spark
# Optional parity against real PySpark (requires Java):
SPARKLESS_TEST_MODE=pyspark uv run pytest tests/spark -m spark

For Airflow orchestrator work:

uv sync --group airflow
uv run pytest tests/orchestration tests/airflow
uv run python examples/airflow_compile.py

The committed toolchain is uv + pytest + ruff + mkdocs.

Commit Messages

Use concise, imperative summaries:

Add typed multi-output step model
Validate SQL dialect capabilities during planning
Document plugin cancellation semantics

Code of Conduct

Be respectful, specific, and constructive. Review the work rather than the person. Assume good intent while asking for evidence on correctness, compatibility, and architecture.

Security Issues

Do not report credential leaks, arbitrary code execution, unsafe reference resolution, or other vulnerabilities through a public issue. Follow the repository security policy.