Problem
Data ingestion pipelines often fail silently. A field goes null, a schema drifts, a source system changes its encoding — and the failure propagates downstream before anyone notices. We needed a way to catch quality issues at the point of ingestion rather than during analytics.
The challenge was scale: pipelines ingested data from dozens of sources with different schemas, cadences, and reliability characteristics. A one-size-fits-all validation approach wouldn't work.
Architecture
The framework is structured around three layers:
Rule Engine — A declarative YAML-based DSL for defining validation rules per dataset. Rules include schema checks, null rate thresholds, referential integrity assertions, and statistical distribution tests. Rules are versioned alongside the pipelines they validate.
Execution Layer — Rules are compiled into Polars or Spark jobs depending on dataset size. Small-to-medium datasets (under ~50GB) use Polars for speed; larger ones use Spark. The executor runs as a Kubernetes Job post-ingestion.
Observability — Validation results are written to a Postgres metadata store. A dashboard exposes per-dataset quality scores, trend lines, and alert history. Failed validations trigger PagerDuty alerts with context about which rules failed and by how much.
Tradeoffs
Polars vs Spark — Polars is significantly faster on single-node workloads, but Spark is necessary for datasets that don't fit in memory. We ended up with a routing layer that picks the right executor based on estimated data size. This adds complexity but is worth it for the performance gains on smaller datasets.
Declarative rules vs code — Keeping rules in YAML makes them accessible to data analysts who aren't Python engineers, but it limits expressiveness. We added an escape hatch for custom Python validators when YAML isn't sufficient, accepting the tradeoff that those validators need code review.
When to fail a pipeline — We distinguish between hard failures (pipeline stops) and soft failures (pipeline continues, issue logged). Misconfiguring this distinction is the most common operational error.
Lessons Learned
Schema drift is the most common failure mode, not data corruption. Building robust schema evolution handling early would have saved significant on-call time.
Validation rules need their own test suite. Rules that are wrong are worse than no rules — they either mask real problems or generate alert fatigue.
Sampling works surprisingly well for catching distribution-level issues at a fraction of the cost. For large datasets, validating a 10% sample catches almost everything except rare edge cases.