Real Apache Airflow questions from data engineering interviews — DAGs, operators, scheduling, backfills, XComs, and executors.
Apache Airflow is the most widely used workflow orchestrator in data engineering, and interviewers expect hands-on knowledge. These questions cover DAG design, operators and sensors, scheduling and the data-interval model, backfills and catchup, idempotency, XComs, connections and hooks, and the different executors. Each question includes a detailed answer grounded in real pipeline experience.
This collection contains 50 curated questions: 26 easy, 6 medium, and 18 hard. There's a strong foundation of fundamentals-focused questions — ideal for building confidence before tackling advanced topics.
The most frequently tested areas in this set are airflow (28), spark (25), partition (16), optimization (14), python (9), and etl (8). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What architecture are you following in your current project, and why?
What are Airflow Operators? Give examples.
Architecturally, how do Job–Stage–Task boundaries in Spark's execution model impact cluster sizing, shuffle cost, and when would you deliberately collapse or split stages?
Explain the concept of checkpointing in Spark and why it is important.
Explain the difference between Azure Data Factory (ADF) and Databricks.
How would you read data from a web API using PySpark?
Tell me about a time when you faced a challenging situation at work and how you handled it.
Triggers in ADF, especially tumbling window triggers.
Walk through the three AQE features in Spark 3.x (coalesce, join switch, skew join)—how they operate at shuffle boundaries, which configs enable them, and what happens when AQE cannot help.
What are the key components of the Spark execution model (Job, Stage, Task)?
What is Azure Data Factory (ADF), and what are its main components?
What is the role of AWS Lambda in a data engineering pipeline?
A data pipeline processes files for different clients stored in separate directories. Explain how you would use dynamic DAG creation to handle client-specific workflows in Airflow.
A task intermittently fails due to external API limitations. How would you configure Airflow retries and alerts to manage this situation efficiently?
Airflow operators, hooks, and scheduler functionality?
Apache Spark Architecture - RDD, DAG, cluster manager, driver node, worker node
Apache Spark Fundamentals - discuss
API calling with Airflow?
Azure Functions vs. Logic Apps?
Cache vs. Persistent storage in Spark?
Can you explain your experience with Docker and Kubernetes?
Can you explain your experience with Jenkins in your project?
Can you give a use case where Delta Live Tables would be ideal?
Can you share an example of a project you worked on that had a significant impact on your organization?
Can you share an experience where you resolved a conflict within your team?
Cloud Composer Overview
Compare ADF vs. Databricks.
Compare Airflow's @daily vs once trigger scheduling.
Connecting BigQuery with Linux
Core services of AWS used in data engineering?
Data Factory vs. Databricks: When to use which?
Data Modeling and Airflow Scheduling - star schema, cron, backfill
Define Airflow and explain it as a workflow orchestration tool.
Defining Tasks in DAG
Deploying DAGs
Describe a project where you implemented a data quality framework.
Describe a real-world use case for using Step Functions with Lambda in a data workflow.
Describe an end-to-end data pipeline project you worked on, highlighting your role and the technologies used.
Describe AWS Glue components and their functions.
Describe how to pass data between tasks in Airflow using XComs.
Describe how to set up retries and timeout for tasks in Cloud Composer.
Describe how you would debug a failing ETL pipeline in production.
Describe how you would monitor ETL job performance and handle long-running tasks.
Describe how you would use AWS Glue to schedule and manage Spark jobs.
Describe the architecture of an ETL pipeline you built in your previous project.
Describe the process and use cases of implementing Azure Data Factory pipelines.
Describe the role of a DAG Scheduler in PySpark
Describe the role of a workflow orchestrator like Airflow in a data pipeline.
Describe the stages of a Spark job and strategies to optimize Spark performance for large datasets.
Describe your monitoring strategy for this pipeline.
Airflow is an open-source platform to author, schedule, and monitor data workflows as code. A workflow is defined as a DAG (Directed Acyclic Graph) — a set of tasks with dependencies and no cycles, so the pipeline always terminates. Each task is an operator; edges define ordering. Airflow schedules DAG runs, tracks each task's state, retries failures, and gives you a UI to observe and re-run pipelines.
An operator is a template for a unit of work (e.g., PythonOperator, BashOperator, SQL operators). A task is an instantiated operator within a DAG. A sensor is a special operator that waits for a condition — a file landing in S3, a partition appearing, or an external job finishing — before downstream tasks run. Using sensors (ideally in 'reschedule' mode) lets pipelines react to upstream data readiness instead of guessing at timing.
Each DAG has a schedule and a data interval — the window a run processes. Tasks should be parameterized by the run's logical date rather than 'now', so they're deterministic. Backfilling re-runs the DAG for past intervals to populate history or recover from a bug; with catchup enabled, Airflow automatically runs missed intervals. Backfills only work safely when tasks are idempotent, so a rerun overwrites rather than duplicates.
XCom (cross-communication) lets tasks pass small pieces of data to each other — for example, a task pushes a file path or row count that a downstream task pulls. XComs are stored in Airflow's metadata database, so they're meant for small values, not large datasets. For big data, pass a pointer (an S3 path or table name) via XCom and let the next task read the actual data from storage.
The executor decides how and where tasks run. SequentialExecutor runs one task at a time (dev only). LocalExecutor runs tasks in parallel on a single machine. CeleryExecutor distributes tasks across a pool of worker machines via a message broker, for horizontal scale. KubernetesExecutor launches each task in its own pod, giving isolation and elastic scaling. Production deployments typically use Celery or Kubernetes executors.
Hands-on SQL, Python, and pipeline courses — build the skills interviewers test.
A structured, resume-boosting certificate covering the full DE stack.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.