DataEngPrep.tech
QuestionsBlogStore
Get PDF Bundle

Interview Questions

Real questions from top companies in Spark/Big Data

700+ Easy450+ Medium650+ Hard
All CategoriesBehavioralSpark/Big DataSQLPython/CodingSystem Design/ArchitectureCloud/ToolsGeneral/Othereasymediumhard
1

What is the difference between repartition and coalesce in Apache Spark?

Spark/Big Datamediumpartitionpythonspark1 min read
BCGCitiDunnhumbyFragma Data Systems+3
→
2

What is the difference between SparkSession and SparkContext in Spark?

Spark/Big Datahardoptimizationpythonspark0.7 min read
AltimetrikAmerican ExpressCitiHexaware+3
→
3

What is the difference between cache() and persist() in Spark? When would you use each?

Spark/Big Datamediumpartitionspark0.7 min read
AccentureCoforgeFreechargeImpetus+1
→
4

What is the difference between groupByKey and reduceByKey in Spark?

Spark/Big Datamediumpartitionspark0.8 min read
AccentureCapcoCoforgeNagarro+1
→
5

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Spark/Big Datamediumjoinpartitionpython0.9 min read
CoforgeDelivery HeroDunnhumbyFragma Data Systems+1
→
6

What strategies can you use to handle skewed data in Spark?

Spark/Big Datamediumjoinpartitionspark0.5 min read
BCGBitwiseCitiHashedIn
→
7

Can you explain the architecture of Apache Spark and its components?

Spark/Big Datahardjoinoptimizationpartition3.2 min read
CoforgeFreechargeNihilent
→
8

Describe the difference between Spark RDDs, DataFrames, and Datasets.

Spark/Big Datahardoptimizationpartitionspark0.5 min read
AccentureFragma Data Systems
→
9

Explain the difference between Spark's map() and flatMap() transformations.

Spark/Big Datamediumpartitionspark0.4 min read
Delivery HeroDunnhumbyFragma Data Systems
→
10

How does Spark's Catalyst Optimizer work? Explain its stages.

Spark/Big Datahardjoinoptimizationspark0.5 min read
DunnhumbyFragma Data SystemsHashedIn
→
11

How do you handle late-arriving data in Spark Structured Streaming?

Spark/Big Datahardsparkwindow0.5 min read
BitwiseIncedoSwiggy
→
12

What is the difference between Managed and External tables in Hive/Spark?

Spark/Big Dataeasyspark0.4 min read
CitiDunnhumbyFragma Data Systems
→
13

What is the small-file problem in Spark, and how do you solve it?

Spark/Big Datahardpartitionspark0.5 min read
Daniel WellingtonIncedoSwiggy
→
14

Explain the concept of Broadcast Join in Spark. When should it be used?

Spark/Big Datamediumjoinsparksql0.4 min read
Delivery HeroDunnhumbyFragma Data Systems
→
15

How do you optimize Spark jobs for better performance? Mention at least 5 techniques.

Spark/Big Datahardjoinoptimizationpartition0.5 min read
Fragma Data SystemsPresidioSwiggy
→
16

Architecturally, how would you justify or challenge Hadoop vs. a cloud-native data lake (S3 + EMR/Databricks) for a greenfield enterprise data platform? Discuss scalability ceilings, cost model trade-offs, and operational complexity.

Spark/Big Datahard
17

When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.

Spark/Big Dataeasy
18

Convert complex SQL (CTEs, window functions, subqueries) to production-grade PySpark. Discuss when to use spark.sql() vs. DataFrame API, and the implications for testability, partitioning, and execution predictability.

Spark/Big Datamedium
19

Design a cost-aware resource strategy for a Databricks workload with spiky and batch jobs. Explain Dynamic Resource Allocation, when to disable it, and how min/max executors and spot instances affect cost and SLAs.

Spark/Big Datahard
20

Design an anti-skew strategy for a join on a high-cardinality key with a long-tail distribution (e.g., a few keys hold 80% of rows). Cover salting, split-skew, AQE, and cost/operational trade-offs.

Spark/Big Datahard

+5 More Questions with Expert Answers

Get the complete 1,800+ question library with detailed, expert-level answers covering SQL, Spark, System Design, Python, Cloud, and Behavioral topics.

Get PDF Bundle — from $21Try Free Sample
123...23Next