Real questions on pipeline design, orchestration, Airflow, DAGs, and workflow scheduling. How data flows from source to destination.
Data pipeline design is central to data engineering. These questions cover orchestration tools (Airflow, Prefect, dbt), DAG design, dependency management, scheduling, idempotency, failure handling, and pipeline architecture at scale. Interviewers probe both theoretical knowledge and real-world implementation experience.
This collection contains 41 curated questions: 8 easy, 15 medium, and 18 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are partition (24), spark (20), join (15), optimization (10), etl (9), and python (6). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
Tell me about yourself and your experience.
What architecture are you following in your current project, and why?
What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.
Explain the differences between Data Warehouse, Data Lake, and Delta Lake
Tell me about your family background
What are Airflow Operators? Give examples.
Briefly introduce yourself and walk us through your journey as a Data Engineer so far.
Can you explain the architecture of Apache Spark and its components?
Explain the types of triggers in ADF, including schedule, tumbling window, and event-based triggers.
Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.
Architecturally, how would you justify or challenge Hadoop vs. a cloud-native data lake (S3 + EMR/Databricks) for a greenfield enterprise data platform? Discuss scalability ceilings, cost model trade-offs, and operational complexity.
Describe the data pipeline architecture you've worked with.
Design a Delta table layout for mixed workload: point lookups by user_id, range scans by date, and full partition scans. Compare partitioning vs. Z-ordering—when to use each, and the rewrite cost trade-off.
Explain Common Table Expressions (CTEs) and their benefits.
Explain strategies for managing schema changes in PySpark over time.
Explain the concept of checkpointing in Spark and why it is important.
Explain the difference between Azure Data Factory (ADF) and Databricks.
Explain the Medallion Architecture (Bronze, Silver, Gold layers).
Explain wide vs. narrow transformations and how they drive shuffle cost, failure domains, and pipeline design. When would you intentionally add a wide transformation, and how do you minimize its impact?
Have you worked on Data Warehousing projects?
How do you handle conflicts within a team? Provide an example.
How do you handle exceptions in Python? Provide an example.
How do you handle memory management in Python?
How would you read data from a web API using PySpark?
How would you read data from a web API? What steps would you follow after reading the data?
Tell me about a time when you faced a challenging situation at work and how you handled it.
Triggers in ADF, especially tumbling window triggers.
What are the key components of AWS Glue, and how do they work together?
What are the key components of the Spark execution model (Job, Stage, Task)?
What challenges did you face, and how did you tackle them?
What is Azure Data Factory (ADF), and what are its main components?
What is normalization and denormalization? When would you use each?
What is the difference between OLTP and OLAP?
What is the purpose of the Bronze, Silver, and Gold layers in a data pipeline?
What is the role of AWS Lambda in a data engineering pipeline?
What would you do if a pipeline failed and you couldn't find the reason?
When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.
Why are you leaving your current company?
Why do you want to join this company?
ADF Optimization Techniques?
Apache Spark Architecture - RDD, DAG, cluster manager, driver node, worker node
A DAG (Directed Acyclic Graph) defines a pipeline as a set of tasks with dependencies and no cycles — it specifies what runs, in what order, and on what schedule. Each node is a task (an operator), and edges define ordering. The 'acyclic' part guarantees the workflow always terminates. Airflow renders the DAG, schedules runs, and tracks each task's state for retries and monitoring.
Schedule with a cron expression or interval and design each run around a data interval (the window it processes). Backfilling re-runs the pipeline for past intervals to populate historical data or recover from a bug — which only works safely if tasks are idempotent and parameterized by the run's logical date rather than 'now'. Airflow's catchup and backfill features automate this.
Configure automatic retries with exponential backoff for transient failures, set alerts (email/Slack) on failure, and route bad records to a dead-letter store instead of failing the whole batch. Make tasks idempotent so a retry can't double-write, use sensors/timeouts to avoid hanging, and design clear task boundaries so a failure resumes from the last successful step rather than restarting everything.
Idempotency means re-running a task with the same input yields the same result, so retries and backfills never create duplicates or corrupt data. Achieve it with partition overwrites, upserts/MERGE on a stable key, and parameterizing by the run's logical date. It's what makes a pipeline safe to retry automatically — a property interviewers probe in almost every pipeline question.
Airflow orchestrates heterogeneous workflows — moving data, triggering Spark jobs, calling APIs — across systems. dbt handles the 'T' in ELT: SQL transformations, testing, and lineage inside the warehouse. Prefect is a modern Python-native orchestrator with a lighter developer experience than Airflow. A common stack uses Airflow (or Prefect) to schedule and dbt to transform.
The Data Engineering Interview Answer Vault bundles 750+ reviewed answers into 7 focused PDF volumes — SQL, Spark, Python, System Design, Cloud, Behavioral, and Data Modeling. Study on any device, no subscription required.
800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.
Turn any topic or your own notes into an interactive, personalized course in 60 seconds.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.