Highlight the tools and technologies you've used in your current project

Our data platform features a modern cloud-native stack built on a Medallion architecture. We leverage Kafka for real-time streams, S3 and Snowflake for tiered storage, Spark/Databricks and dbt for robust processing, and Airflow for orchestration, ensuring scalable and high-quality data delivery.

Mechanics and Rationale

* Ingestion & Storage: Fivetran and Airbyte handle managed Change Data Capture (CDC) ingestion from various sources into S3, our cost-effective data lake (often in Delta Lake format). For high-throughput event streams, Kafka captures application logs and user interactions, with consumer groups ensuring reliable, at-least-once delivery by managing offsets. Snowflake then serves as our analytical data warehouse, optimized for complex SQL queries and BI, leveraging its automatic micro-partitioning and clustering for efficient query performance.
* Processing & Orchestration: Spark on Databricks performs large-scale, distributed ETL and feature engineering, handling complex transformations that benefit from its distributed processing capabilities (e.g., managing data shuffling and partitions). dbt is critical for defining analytical transformations within Snowflake, promoting SQL best practices, data quality testing, documentation, and lineage tracking. I've contributed to establishing our team's dbt standards for model structure and testing. Airflow orchestrates our entire data pipeline, managing Directed Acyclic Graphs (DAGs) for daily batch jobs, dependencies, retries, and alerting on failures.
* BI, Observability & IaC: Tableau and Looker provide interactive dashboards and a semantic layer for consistent metric definitions to business users. Datadog offers comprehensive observability, monitoring pipeline health, resource utilization, and alerting on anomalies. Terraform manages our cloud infrastructure (Infrastructure as Code), while GitLab CI/CD automates code deployment, testing, and infrastructure changes, enabling rapid and reliable delivery.

Concrete Example: Medallion Architecture Flow

Our Medallion architecture organizes data into Bronze (raw), Silver (cleaned, conformed), and Gold (business-ready, aggregated) layers. For instance, raw Kafka events land in S3 (Bronze). Spark then processes these into structured Delta Lake tables (Silver), handling schema evolution and deduplication by leveraging Delta Lake's transaction log. Finally, dbt transforms and aggregates this Silver layer data into Snowflake Gold tables, ready for BI consumption, ensuring data quality improves at each stage.

-- models/gold/dim_customer.sql
{{ config(materialized='table', unique_key='customer_id') }}
SELECT
    s.customer_id,
    s.first_name,
    s.last_name,
    s.email,
    COUNT(o.order_id) AS total_orders,
    SUM(o.order_total) AS total_spend
FROM {{ ref('silver_customers') }} s
LEFT JOIN {{ ref('silver_orders') }} o ON s.customer_id = o.customer_id
GROUP BY 1, 2, 3, 4

In the interview, also mention…

In the interview, also mention specific challenges you've solved with these tools, like optimizing a slow-running Spark job or ensuring data quality with dbt tests.