Medium-level spark & big data questions from real data engineering interviews.
These medium spark & big data questions are selected from real interviews at top companies. Each question includes a detailed expert answer and pro tip to help you nail your interview. This set leans toward the medium-difficulty band most real interviews actually live in (49 of 49). Recurring themes are spark, partition, and join — these patterns appear most often in real interviews and reward the deepest preparation. These questions have been reported across 40 companies including Infosys and Altimetrik. Average answer is around 1 minute of reading — plan roughly 1 hour to work through the full set thoughtfully.
This collection contains 49 curated questions: 0 easy, 49 medium. The balanced mix of difficulties makes this set suitable for engineers at any career stage.
The most frequently tested areas in this set are spark (44), partition (42), join (21), sql (17), python (12), and window (3). Focusing on these topics will give you the highest return on your preparation time.
Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What is the difference between repartition and coalesce in Apache Spark?
What is the difference between cache() and persist() in Spark? When would you use each?
What is the difference between groupByKey and reduceByKey in Spark?
What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.
What strategies can you use to handle skewed data in Spark?
Explain the difference between Spark's map() and flatMap() transformations.
Explain the concept of Broadcast Join in Spark. When should it be used?
Convert complex SQL (CTEs, window functions, subqueries) to production-grade PySpark. Discuss when to use spark.sql() vs. DataFrame API, and the implications for testability, partitioning, and execution predictability.
Explain how Adaptive Query Execution changes the economics of Spark tuning. What problems does it solve at runtime, and when might you still need manual intervention (e.g., salting, broadcast hints)?
Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.
Explain strategies for managing schema changes in PySpark over time.
How do you drop columns with null values in PySpark?
How do you handle data skewness in Spark?
How would you read data from a web API using PySpark?
What is Adaptive Query Execution (AQE) in Spark 3.x, and how does it improve performance?
What is the difference between repartition and coalesce in Spark?
When and how do you use Broadcast Join in Spark?
What is broadcasting in Spark, and why is it used? Can you give an example of its use?
What is the difference between map and flatMap in Spark, and when would you use each?
What is the purpose of the Bronze, Silver, and Gold layers in a data pipeline?
What work is done by the executor memory in Spark?
When and how do you use Broadcast Join?
Write a Python script to find the count of each word in a text file using Spark.
Write the PySpark code to find the second highest salary in each department.
Broadcast Joins and Shuffle Merge Joins?
Can you explain the concept of mappers in Spark, and how are they used in data transformations?
Code a simple PySpark job to read a JSON file, filter records, and write output in Parquet format.
How do you handle out-of-memory errors in Spark jobs?
How do you handle very large datasets in Spark to ensure scalability and efficiency?
Provide specific examples of challenges faced with PySpark and SQL and solutions implemented.
Split a DataFrame such that even numbers appear in one column and odd numbers in another
Steps to mount storage in Databricks.
Transformation vs. Action in PySpark?
What Hadoop command would you use to merge multiple files into one?
What are the limitations of the REORG command with respect to large datasets?
What causes Out of Memory (OOM) issues in Databricks, and how do you resolve them?
What is Shuffle and How to Handle It in Spark
Write PySpark code to extract data from a CSV and create a table.
Write PySpark code to save a DataFrame in Parquet format to an S3 bucket.
Write a PySpark job that calculates the number of unique users who logged in per day, but exclude any logins from inactive users listed in a separate file.
Write a PySpark script to filter out invalid records from a dataset and calculate the average for a specific column, ensuring the schema is strictly defined at runtime.
Write a PySpark script to process data stored in Delta format and transform it into Parquet.
Write a PySpark script to read a CSV file, filter rows where the age column is less than 18, and write the result to a new CSV file.
Write a complete PySpark program from import statements to the stop statement, covering transformations and actions.
Write a transformation in PySpark to join and clean multiple raw input sources
Write code to read data from Delta Lake in S3 and perform upsert based on primary key
Write maintainable, efficient Pandas or PySpark code.
Your Kafka producer schema has changed, and the new data includes additional fields. How would you ensure backward compatibility using Schema Registry while consuming data from the same topic?
Z-Ordering - use cases for partitioned Delta tables
The Data Engineering Interview Answer Vault bundles 750+ reviewed answers into 7 focused PDF volumes — SQL, Spark, Python, System Design, Cloud, Behavioral, and Data Modeling. Study on any device, no subscription required.
800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.
Turn any topic or your own notes into an interactive, personalized course in 60 seconds.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.