Skip to content

Distribution

Distribution describes how ETLantic plugins are packaged, published, discovered, installed, and maintained across the ecosystem.

A healthy plugin ecosystem depends on stable distribution practices. Plugins should be independently releasable while remaining compatible with the ETLantic Plugin SDK and supported standards.

Goals

Plugin distribution should:

  • Support independent releases.
  • Encourage semantic versioning.
  • Make installation simple.
  • Preserve compatibility metadata.
  • Enable automated discovery.
  • Support community and commercial plugins.

Philosophy

The ETLantic core should remain small.

Functionality grows through independently distributed plugins.

ETLantic Core
        │
        ▼
 Python Package Index
        │
 ┌──────┼─────────────────────────┐
 ▼      ▼         ▼              ▼
Execution Dataframe Storage   Resource
 Plugins   Plugins  Plugins   Providers

Packaging

Plugins should be distributed as standard Python packages.

Recommended naming convention:

  • etlantic-airflow
  • etlantic-polars
  • etlantic-pandas
  • etlantic-postgres
  • etlantic-s3
  • etlantic-keyring
  • etlantic-secrets-aws
  • etlantic-secrets-azure
  • etlantic-secrets-gcp
  • etlantic-secrets-vault

Third-party plugins should follow a consistent naming convention where practical.

Installation

Typical installation:

pip install etlantic-airflow

Multiple plugins may be installed simultaneously.

Profiles determine which plugins are used.

Discovery

Installed plugins are discoverable through domain-specific helpers (no global registry class):

from etlantic.dataframe import discover_dataframe_plugins
from etlantic.orchestration import discover_orchestrator_plugins
from etlantic.spark import discover_spark_plugins, discover_spark_providers
from etlantic.sql import discover_sql_plugins

dataframe_plugins = discover_dataframe_plugins()
sql_plugins = discover_sql_plugins()
spark_plugins = discover_spark_plugins()
spark_providers = discover_spark_providers()
orchestrators = discover_orchestrator_plugins()

CLI: etlantic plugin list. Discovery does not require manual registration in normal deployments. Secret providers are attached to the runtime / profile.

Compatibility Metadata

Every plugin should publish:

  • Plugin name
  • Plugin version
  • Supported ETLantic version(s)
  • Supported SDK version
  • Supported ODCS version(s)
  • Supported DTCS version(s)
  • Supported DPCS version(s)
  • Capability declarations

Planning uses this metadata when selecting plugins.

Semantic Versioning

Plugins should follow Semantic Versioning.

  • MAJOR — breaking SDK or behavior changes
  • MINOR — backward-compatible features
  • PATCH — bug fixes and non-semantic improvements

Release Workflow

Recommended workflow:

  1. Implement
  2. Test
  3. Run SDK conformance suite
  4. Publish package
  5. Publish release notes
  6. Update compatibility matrix

Dependency Management

Plugins should minimize dependencies and isolate backend-specific libraries.

Optional dependencies should be preferred where practical to avoid unnecessary install size for users who do not need every backend.

Security

Plugin authors should:

  • Sign releases when possible.
  • Keep dependencies updated.
  • Avoid embedding credentials.
  • Follow secure disclosure practices.
  • Publish supported version ranges.
  • Ensure secret-provider packages pass redaction, rotation, lease, and sentinel-leak conformance tests.

Best Practices

  • Keep plugins independently releasable.
  • Publish clear compatibility metadata.
  • Use semantic versioning.
  • Ship documentation with every release.
  • Maintain changelogs.

Anti-Patterns

Avoid:

  • Bundling all plugins into the core package.
  • Publishing plugins without compatibility metadata.
  • Introducing breaking changes in minor releases.
  • Hiding runtime dependencies.
  • Depending on internal ETLantic APIs.

Key Principle

ETLantic grows through a distributed ecosystem of independently versioned plugins. Stable packaging, discovery, and compatibility metadata allow users to compose the execution environment that best fits their pipelines.

Next Step

Continue with the Examples section to see plugins used in complete pipelines.