The PySpark questions that show up in real data engineering interviews — DataFrame API, UDFs, partitioning, and performance tuning.
PySpark is the most common way data engineers write Spark in production, and interviewers test it deeply. These questions cover the DataFrame API, transformations vs actions and lazy evaluation, when (and when not) to use UDFs vs pandas UDFs, partitioning and shuffle mechanics, broadcast joins, caching, and real-world performance tuning. Each question includes a detailed answer with code.
This collection contains 29 curated questions: 4 easy, 11 medium, and 14 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are spark (29), partition (20), python (17), sql (17), optimization (9), and join (6). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What is the difference between repartition and coalesce in Apache Spark?
What is the difference between SparkSession and SparkContext in Spark?
What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.
Describe the difference between Spark RDDs, DataFrames, and Datasets.
How does Spark's Catalyst Optimizer work? Explain its stages.
Convert complex SQL (CTEs, window functions, subqueries) to production-grade PySpark. Discuss when to use spark.sql() vs. DataFrame API, and the implications for testability, partitioning, and execution predictability.
Explain strategies for managing schema changes in PySpark over time.
Explain the difference between Azure Data Factory (ADF) and Databricks.
How do you drop columns with null values in PySpark?
How do you optimize Spark jobs for performance?
How would you read data from a web API using PySpark?
Implement a Spark job to find the top 10 most frequent words in a large text file.
Retrieve the most recent sale_timestamp for each product (Latest Transaction).
What is the difference between Spark RDDs, DataFrames, and Datasets?
What work is done by the executor memory in Spark?
When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.
Why is SparkSession used in Spark 2.0 and later versions?
Write a Python script to find the count of each word in a text file using Spark.
Write the PySpark code to find the second highest salary in each department.
Add Row Numbers using window function in PySpark
Code a simple PySpark job to read a JSON file, filter records, and write output in Parquet format.
Command to Read JSON Data and Options
Count occurrences of a specific word in a file
Create a DataFrame with default column types
Create Spark Session, read CSV, join, and write as table. Provide example code.
Discuss the tech stacks and responsibilities at Morgan Stanley
Explain database drivers/connectors and their use cases.
Explain Delta Live Tables and their features, such as declarative pipeline definition and automatic data validation.
Explain PySpark's Catalyst Optimizer.
PySpark is the Python API for Apache Spark — it lets you write Spark jobs in Python instead of Scala. Under the hood, DataFrame operations are translated into the same Catalyst-optimized JVM execution plan, so PySpark DataFrame code runs at near-Scala speed. The gap appears only with Python UDFs, which serialize data between the JVM and Python workers. PySpark is the default choice for data engineering because of Python's ecosystem.
An RDD is a low-level, unstructured distributed collection with no schema and no query optimization. A DataFrame is a higher-level, schema-aware abstraction that runs through the Catalyst optimizer and Tungsten execution engine, making it much faster and more memory-efficient. In PySpark you should almost always use DataFrames; drop to RDDs only for low-level control the DataFrame API can't express.
A regular Python UDF forces Spark to serialize each row from the JVM to a Python process, run your function, and serialize the result back — row by row, bypassing Catalyst optimization. Prefer built-in Spark SQL functions whenever possible. When you must use Python logic, use pandas UDFs (vectorized UDFs), which process data in Arrow-backed batches and are dramatically faster than row-at-a-time UDFs.
Start by reading the Spark UI to find the bottleneck stage. Common fixes: minimize shuffles (broadcast small tables, pre-partition on join keys), handle skew with Adaptive Query Execution or salting, cache reused DataFrames, avoid Python UDFs in favor of native/pandas UDFs, prune columns early with select, push filters down, and write output in partitioned columnar formats like Parquet to avoid the small-file problem.
PySpark transformations (select, filter, withColumn, join) are lazy — they build a logical plan but do not execute. Execution only happens when you call an action (show, count, collect, write). Laziness lets Catalyst optimize the whole pipeline at once — combining filters, pruning columns, and reordering joins — rather than running each step blindly. It's why chaining many transformations is cheap until the first action triggers the job.
The Data Engineering Interview Answer Vault bundles 750+ reviewed answers into 7 focused PDF volumes — SQL, Spark, Python, System Design, Cloud, Behavioral, and Data Modeling. Study on any device, no subscription required.
800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.
Turn any topic or your own notes into an interactive, personalized course in 60 seconds.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.