Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Coalesce reduces partitions by merging (no full shuffle); repartition redistributes (full shuffle). Coalesce: df.coalesce(4)—can only...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Coalesce reduces partitions by merging (no full shuffle); repartition redistributes (full shuffle). Coalesce: df.coalesce(4)—can only decrease. Repartition: df.repartition(8) or df.repartition('key'). Use coalesce to reduce (e.g., before write); repartition to increase or change distribution. Coalesce avoids shuffle; repartition guarantees even distribution. Best practice: coalesce after filter to reduce output files; repartition when more parallelism needed.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Repartition to reduce. Pro-Move: 'coalesce = no shuffle; repartition = full shuffle.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.