The PySpark questions that show up in real data engineering interviews — DataFrame API, UDFs, partitioning, and performance tuning.
PySpark is the most common way data engineers write Spark in production, and interviewers test it deeply. These questions cover the DataFrame API, transformations vs actions and lazy evaluation, when (and when not) to use UDFs vs pandas UDFs, partitioning and shuffle mechanics, broadcast joins, caching, and real-world performance tuning. Each question includes a detailed answer with code.
This collection contains 50 curated questions: 11 easy, 13 medium, and 26 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are spark (50), partition (27), python (23), sql (23), optimization (20), and join (10). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What is the difference between repartition and coalesce in Apache Spark?
What is the difference between SparkSession and SparkContext in Spark?
What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.
Describe the difference between Spark RDDs, DataFrames, and Datasets.
How does Spark's Catalyst Optimizer work? Explain its stages.
Convert complex SQL (CTEs, window functions, subqueries) to production-grade PySpark. Discuss when to use spark.sql() vs. DataFrame API, and the implications for testability, partitioning, and execution predictability.
Explain strategies for managing schema changes in PySpark over time.
Explain the difference between Azure Data Factory (ADF) and Databricks.
How do you drop columns with null values in PySpark?
How do you optimize Spark jobs for performance?
How would you read data from a web API using PySpark?
Implement a Spark job to find the top 10 most frequent words in a large text file.
Retrieve the most recent sale_timestamp for each product (Latest Transaction).
What is the difference between Spark RDDs, DataFrames, and Datasets?
What work is done by the executor memory in Spark?
When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.
Why is SparkSession used in Spark 2.0 and later versions?
Write a Python script to find the count of each word in a text file using Spark.
Write the PySpark code to find the second highest salary in each department.
A JSON file with evolving schema needs to be ingested into a DataFrame. How would you handle new fields dynamically in PySpark without breaking the job for previous structures?
Add Row Numbers using window function in PySpark
Apache Spark Fundamentals - discuss
Basic Spark commands – Create RDD, Load data, Filter
Cache vs. Persistent storage in Spark?
Cache() vs Persist(): Explain the difference and use cases for caching and persisting data in Spark with memory levels.
Case Class and StructType Syntax
Code a simple PySpark job to read a JSON file, filter records, and write output in Parquet format.
Concatenate Columns in PySpark
Count occurrences of a specific word in a file
Count occurrences of each character in a string
Create a DataFrame with default column types
Create Spark Session, read CSV, join, and write as table. Provide example code.
Data Factory vs. Databricks: When to use which?
Databricks vs. PySpark?
Define what a User-Defined Function (UDF) is and how to register it in PySpark.
Describe AWS Glue components and their functions.
Describe how you would optimize a join between two large tables where one is significantly smaller, using broadcast joins in PySpark.
Describe how you would optimize slow-running Spark jobs in a distributed environment.
Describe how you would use PySpark to aggregate and summarize large transaction datasets.
Describe the role of a DAG Scheduler in PySpark
Difference between var, val, and def in Scala
Discuss common transformations used in Spark code.
Discuss the tech stacks and responsibilities at Morgan Stanley
Duplicate characters in a string (e.g., '123a!' to '112233aa!!').
Explain aggregation functions in PySpark with examples and use cases.
Explain database drivers/connectors and their use cases.
Explain Delta Live Tables and their features, such as declarative pipeline definition and automatic data validation.
Explain how to overwrite a file stored in S3 using PySpark.
Explain PySpark's Catalyst Optimizer.
Explain SCD1 and SCD2 in Databricks PySpark with examples.
PySpark is the Python API for Apache Spark — it lets you write Spark jobs in Python instead of Scala. Under the hood, DataFrame operations are translated into the same Catalyst-optimized JVM execution plan, so PySpark DataFrame code runs at near-Scala speed. The gap appears only with Python UDFs, which serialize data between the JVM and Python workers. PySpark is the default choice for data engineering because of Python's ecosystem.
An RDD is a low-level, unstructured distributed collection with no schema and no query optimization. A DataFrame is a higher-level, schema-aware abstraction that runs through the Catalyst optimizer and Tungsten execution engine, making it much faster and more memory-efficient. In PySpark you should almost always use DataFrames; drop to RDDs only for low-level control the DataFrame API can't express.
A regular Python UDF forces Spark to serialize each row from the JVM to a Python process, run your function, and serialize the result back — row by row, bypassing Catalyst optimization. Prefer built-in Spark SQL functions whenever possible. When you must use Python logic, use pandas UDFs (vectorized UDFs), which process data in Arrow-backed batches and are dramatically faster than row-at-a-time UDFs.
Start by reading the Spark UI to find the bottleneck stage. Common fixes: minimize shuffles (broadcast small tables, pre-partition on join keys), handle skew with Adaptive Query Execution or salting, cache reused DataFrames, avoid Python UDFs in favor of native/pandas UDFs, prune columns early with select, push filters down, and write output in partitioned columnar formats like Parquet to avoid the small-file problem.
PySpark transformations (select, filter, withColumn, join) are lazy — they build a logical plan but do not execute. Execution only happens when you call an action (show, count, collect, write). Laziness lets Catalyst optimize the whole pipeline at once — combining filters, pruning columns, and reordering joins — rather than running each step blindly. It's why chaining many transformations is cheap until the first action triggers the job.
Hands-on SQL, Python, and pipeline courses — build the skills interviewers test.
A structured, resume-boosting certificate covering the full DE stack.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.