Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why skew matters**: Uneven partition sizes = stragglers; one partition 10x others = 90% resources idle. **Resolutions**: (1) **Salting**—add random suffix to key, redistribute, aggregate, then merge. (2) **Broadcast**—small dimension, avoid shuffle. (3) **Split hot...
Why skew matters: Uneven partition sizes = stragglers; one partition 10x others = 90% resources idle. Resolutions: (1) Salting—add random suffix to key, redistribute, aggregate, then merge. (2) Broadcast—small dimension, avoid shuffle. (3) Split hot keys—e.g., user_id 12345 → 12345_0, 12345_1. (4) AQE—Spark 3.x can detect skew. Scalability trade-offs: Salting adds complexity; broadcast has size limit. Cost implications: Skew = 2–5x wasted compute; fixing = significant savings. Best practice: Identify via Spark UI (task duration variance); apply salting; monitor partition sizes.
Red Flag: 'We don't have skew'—almost every production system has some. Pro-Move: 'Spark UI showed 1 partition 50x others; salting cut job from 2hr to 20min.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.