Describe a project where you implemented a data quality framework.

I led a project to implement a robust data quality framework, primarily leveraging Great Expectations and dbt tests, integrated into our Airflow pipelines. This initiative successfully reduced data-related errors in critical business reports by 80%.

The core mechanics involved establishing a "shift-left" approach to data quality. We used Great Expectations to define data contracts and validate raw ingested data, ensuring schema conformity, data type integrity, and freshness checks at the source. This proactive validation step, executed as an Airflow task, would halt downstream processing if critical expectations failed, preventing bad data from propagating. Subsequently, dbt tests were implemented within our transformation layer, running directly on our Snowflake data warehouse. These tests enforced business rules, uniqueness constraints, referential integrity across models, and identified anomalies in data distributions.

For instance, a common issue was NULL values appearing in a customer_id column due to an upstream API change. Our Great Expectations suite caught this schema violation at ingestion, preventing the data from reaching our dbt models. Later, a dbt not_null test on the transformed dim_customers model would have also caught this, ensuring data quality at multiple stages. Automated alerts via Slack and PagerDuty were configured for any test failures, enabling rapid root cause analysis and resolution by the data engineering team.

-- Example dbt test in models/marts/core/dim_customers.yml
models:
  - name: dim_customers
    columns:
      - name: customer_id
        tests:
          - not_null
          - unique
      - name: email
        tests:
          - dbt_utils.not_null_where:
              where_clause: "is_active = TRUE"

In the interview, also mention the importance of data ownership and how defining these expectations fosters better collaboration with data producers.