Real Apache Airflow questions from data engineering interviews — DAGs, operators, scheduling, backfills, XComs, and executors.
Apache Airflow is the most widely used workflow orchestrator in data engineering, and interviewers expect hands-on knowledge. These questions cover DAG design, operators and sensors, scheduling and the data-interval model, backfills and catchup, idempotency, XComs, connections and hooks, and the different executors. Each question includes a detailed answer grounded in real pipeline experience.
This collection contains 21 curated questions: 5 easy, 6 medium, and 10 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are partition (13), spark (13), airflow (12), optimization (9), join (4), and etl (3). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What architecture are you following in your current project, and why?
What are Airflow Operators? Give examples.
Architecturally, how do Job–Stage–Task boundaries in Spark's execution model impact cluster sizing, shuffle cost, and when would you deliberately collapse or split stages?
Explain the concept of checkpointing in Spark and why it is important.
Explain the difference between Azure Data Factory (ADF) and Databricks.
How would you read data from a web API using PySpark?
Tell me about a time when you faced a challenging situation at work and how you handled it.
Triggers in ADF, especially tumbling window triggers.
Walk through the three AQE features in Spark 3.x (coalesce, join switch, skew join)—how they operate at shuffle boundaries, which configs enable them, and what happens when AQE cannot help.
What are the key components of the Spark execution model (Job, Stage, Task)?
What is Azure Data Factory (ADF), and what are its main components?
What is the role of AWS Lambda in a data engineering pipeline?
Apache Spark Architecture - RDD, DAG, cluster manager, driver node, worker node
Can you share an example of a project you worked on that had a significant impact on your organization?
Compare Airflow's @daily vs once trigger scheduling.
Connecting BigQuery with Linux
Describe a project where you implemented a data quality framework.
Describe an end-to-end data pipeline project you worked on, highlighting your role and the technologies used.
Describe how you would debug a failing ETL pipeline in production.
Describe the architecture of an ETL pipeline you built in your previous project.
Describe your monitoring strategy for this pipeline.
Airflow is an open-source platform to author, schedule, and monitor data workflows as code. A workflow is defined as a DAG (Directed Acyclic Graph) — a set of tasks with dependencies and no cycles, so the pipeline always terminates. Each task is an operator; edges define ordering. Airflow schedules DAG runs, tracks each task's state, retries failures, and gives you a UI to observe and re-run pipelines.
An operator is a template for a unit of work (e.g., PythonOperator, BashOperator, SQL operators). A task is an instantiated operator within a DAG. A sensor is a special operator that waits for a condition — a file landing in S3, a partition appearing, or an external job finishing — before downstream tasks run. Using sensors (ideally in 'reschedule' mode) lets pipelines react to upstream data readiness instead of guessing at timing.
Each DAG has a schedule and a data interval — the window a run processes. Tasks should be parameterized by the run's logical date rather than 'now', so they're deterministic. Backfilling re-runs the DAG for past intervals to populate history or recover from a bug; with catchup enabled, Airflow automatically runs missed intervals. Backfills only work safely when tasks are idempotent, so a rerun overwrites rather than duplicates.
XCom (cross-communication) lets tasks pass small pieces of data to each other — for example, a task pushes a file path or row count that a downstream task pulls. XComs are stored in Airflow's metadata database, so they're meant for small values, not large datasets. For big data, pass a pointer (an S3 path or table name) via XCom and let the next task read the actual data from storage.
The executor decides how and where tasks run. SequentialExecutor runs one task at a time (dev only). LocalExecutor runs tasks in parallel on a single machine. CeleryExecutor distributes tasks across a pool of worker machines via a message broker, for horizontal scale. KubernetesExecutor launches each task in its own pod, giving isolation and elastic scaling. Production deployments typically use Celery or Kubernetes executors.
The Data Engineering Interview Answer Vault bundles 750+ reviewed answers into 7 focused PDF volumes — SQL, Spark, Python, System Design, Cloud, Behavioral, and Data Modeling. Study on any device, no subscription required.
800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.
Turn any topic or your own notes into an interactive, personalized course in 60 seconds.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.