Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why these concepts matter**: They drive most Spark performance issues. **Shuffle**: Redistributes data; expensive (network, serialization). Reduce via broadcast, partition pruning, avoid unnecessary groupBy/join. **Skew**: Uneven partition sizes; stragglers. Resolve: salting,...
Why these concepts matter: They drive most Spark performance issues. Shuffle: Redistributes data; expensive (network, serialization). Reduce via broadcast, partition pruning, avoid unnecessary groupBy/join. Skew: Uneven partition sizes; stragglers. Resolve: salting, split hot keys, broadcast small side. Caching: Persist hot data; unpersist when done. Trade-off: memory vs recompute. Scalability trade-offs: Shuffle = O(data); skew = 1 partition slows all; cache = memory. Cost implications: Shuffle and skew = 2–10x cost; cache = faster when reused. Best practice: Minimize shuffle; detect skew (Spark UI); cache strategically.
Red Flag: Caching without measuring reuse. Pro-Move: 'Spark UI showed skew; salting + broadcast—runtime 2hr to 20min.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.