Data quality frameworks

Intuition

Data profiling is a one-time investigation: what does this data look like, right now. A data quality framework turns the same questions — null rate, cardinality, range, referential integrity — into assertions that run automatically, every time, and fail loudly the moment reality stops matching them.

The reason this needs to be a distinct discipline rather than “just write a test” is where the assertion runs matters as much as what it checks. A check that runs after a bad load has already reached a downstream table has already let the damage propagate; the same check running before the load commits prevents it. Every framework here is really answering the same question — “where in the pipeline does the check fire, and what happens when it fails” — differently.

Mechanics

dbt tests: assertions colocated with the transformation that produces the data

schema.yml

models:
  - name: fact_orders
    columns:
      - name: order_id
        data_tests: [unique, not_null]
      - name: customer_id
        data_tests:
          - not_null
          - relationships:
              to: ref('dim_customer')
              field: customer_key
      - name: status
        data_tests:
          - accepted_values:
              values: ['complete', 'pending', 'refunded', 'cancelled']

Each test compiles to a SQL query that returns zero rows on success and one row per violation on failure — unique and not_null are self-explanatory; relationships checks referential integrity exactly like the orphaned foreign key check from the profiling page, but as a standing assertion instead of a one-time query; accepted_values catches an unexpected new status value the moment it appears, rather than silently flowing through every downstream aggregate grouped by status.

These run as part of dbt build or dbt test, colocated with the model that produces the data — the person who changes the transformation logic is the person most likely to see the test fail, immediately, in the same run.

Data contracts: failing before the query runs, not after

models:
  - name: fact_orders
    config:
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: integer
      - name: amount_cents
        data_type: integer

A dbt data test runs after a model builds and checks the result. A data contract is enforced at compile time — dbt checks the model’s actual output schema against the declared contract before running, and refuses to build if they disagree. This catches a broken upstream schema change (a column renamed, a type changed from integer to text) before a single row is written to the target table, rather than after — the difference between “the pipeline failed to build” and “the pipeline built successfully and quietly produced garbage that a test caught an hour later, after other jobs had already read from it.”

Great Expectations: profiling and validation as one artifact

import great_expectations as gx

validator.expect_column_values_to_not_be_null("customer_id")
validator.expect_column_values_to_be_between("amount_cents", min_value=0, max_value=1_000_000)
validator.expect_column_distinct_values_to_be_in_set(
    "status", ["complete", "pending", "refunded", "cancelled"]
)

The distinguishing idea in Great Expectations is that an “expectation” is both a validation rule and a self-documenting statement about what the data should look like — the same object that fails a pipeline run on violation also generates a human-readable data quality report. It’s commonly used independent of dbt, at ingestion time or on raw data before any transformation happens — checking the data as it arrives, rather than checking a transformation’s output.

Where each check should actually run

The meaningful design decision isn’t “which tool” — it’s at which stage does a given check run, because that determines the blast radius of a failure:

Stage	What a failure prevents
At ingestion (Great Expectations, or equivalent)	Bad data entering the warehouse at all
At build time (dbt contracts)	A model building with the wrong schema
Post-build (dbt tests)	Bad data being trusted downstream, after it already landed

A check that only runs post-build has already let bad data land in a table that other jobs, dashboards, or models may read from before the test suite finishes running — the check still catches the problem, just after some of the damage may already be visible to a consumer.

Cost & limits

Every test adds pipeline run time, and a large dbt project can accumulate hundreds of tests whose cumulative execution time becomes a real part of the pipeline’s total duration — this is a genuine trade against faster iteration, not a free safety net, and it’s common to run a fast subset of tests (uniqueness, not-null on primary keys) on every run and a fuller suite on a schedule.

Data contracts are strict by construction — a contract enforces an exact schema match, so an intentional schema change (adding a column) requires updating the contract in the same change, or the build fails. This is the point (catching unintentional drift), but it means contracts add friction to legitimate, planned schema evolution too, and that friction is a cost worth accepting deliberately rather than discovering by surprise.

When NOT to use it

Do not add a data contract to a model still under active, frequent schema iteration. Contracts are strongest once a model’s schema has stabilized and unplanned drift is the main risk being guarded against — applied too early, they mostly just add friction to expected, frequent changes.

Do not rely solely on post-build dbt tests for data that other systems consume before the test suite completes. If a downstream dashboard or model reads from a table the moment it’s built, rather than after the full test suite passes, a post-build-only testing strategy has a real window where bad data is visible before it’s caught — for that data, an ingestion-time check or a contract catches the problem earlier.

Do not write an accepted_values test with a list you expect to change frequently without a plan for maintaining it. A status enum that legitimately gains new values periodically needs either a broader maintenance process for the test itself, or a different kind of check (logging new values for review, rather than hard-failing on them) — otherwise the test becomes a recurring, expected failure that erodes trust in test failures generally.

Real-world usage

Modern dbt-based analytics engineering teams run data_tests as a required step in CI before merging any model change, catching regressions before they reach production — a broken not_null test on a primary key blocks the pull request rather than blocking a 3am pipeline run days later. Ingestion-time validation (Great Expectations or similar) is more common at the boundary where external, less-trusted data enters an organization’s systems — a vendor feed, a partner API, a user upload — where the incoming data’s shape can’t be guaranteed by anything the organization controls.

Failure modes

The dashboard that showed bad numbers for six hours because the only check was post-build. A pipeline architecture where tests run after the data lands and other jobs read immediately has a real gap during which bad data is live and visible — symptom: a data quality incident where the test suite “worked” (it eventually caught the problem) and a stakeholder still saw wrong numbers before it did.

The contract that blocks a legitimate deploy nobody remembered to update. A schema contract left unexpectedly in place after an intentional change (a new column added to the model but not the contract) fails the build for a reason that looks like a bug rather than a maintenance gap — symptom: a build failure that requires reading the contract’s diff against the model’s actual output to understand, rather than being obvious from the error alone if the two aren’t reviewed together.

The test suite so slow that it gets skipped under deadline pressure. Hundreds of dbt tests accumulated without pruning or tiering (fast tests every run, slow tests nightly) can make the full suite slow enough that teams start skipping it under time pressure — the framework’s value degrades to whatever fraction of it people actually run when it’s inconvenient not to.

Practice problems

1. A dbt model has a not_null test on customer_id that starts failing. Where in the pipeline did the check run, and what does that mean for whether bad data already reached other consumers?

Post-build (a data_tests assertion), meaning the model already built and the null values already landed in the target table before the test ran and failed — any job or dashboard reading from that table before the test suite completed could have already seen the bad data. A contract or ingestion-time check would have caught it earlier.

2. A team wants to add a new refunded_partial status value to their orders table. Their dbt accepted_values test on status starts failing. Is this a data quality bug?

No — it’s the test working as designed, catching an intentional schema change that the test itself hasn’t been updated to expect. The fix is updating the accepted_values list alongside the change that introduces the new status, in the same pull request.

3. Why might a team choose to enforce a data contract on a model that already has full dbt test coverage (uniqueness, not-null, relationships)?

Tests catch schema and data problems after the model builds; a contract catches a schema mismatch at build time, before the (potentially expensive) transformation runs and before any bad-schema output can be read by anything downstream. For a model with high fan-out to many downstream consumers, the earlier failure point is worth the added strictness.

Check yourself

A dbt model's schema silently drifts from what downstream models expect -- a column's type changes upstream. What's the key difference between catching this with a data test versus a data contract?

• There is no difference -- both catch it at the same point
• A data test catches it after the model has already built with the wrong schema; a contract fails the build before it runs at all
• Data contracts can only check for NULL values
• Tests are more strict than contracts

Interview answers

“How would you catch data quality issues in a pipeline?” Layer checks at different stages depending on blast radius: ingestion-time validation for external data before it enters the warehouse, schema contracts at build time to catch structural drift before a transformation runs, and post-build tests (uniqueness, not-null, referential integrity, accepted values) as a final safety net. The caveat: which stage a check runs at determines how much damage happens before it’s caught — a post-build-only strategy still catches problems, just after some downstream consumers may have already read the bad data.

“What’s the difference between a dbt test and a dbt contract?” A test is a SQL assertion that runs after a model builds and checks its output data; a contract is a schema declaration enforced at compile time, before the model runs, checking structure rather than content. The caveat that shows real use: contracts add real friction to legitimate schema changes — they need to be updated in the same change that intentionally alters a model’s output shape, or the build fails for a reason that looks like a bug rather than a missed update.