Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Skew occurs when a few keys hold disproportionate data, causing hotspot tasks and stragglers. **Why it matters**: One task taking 10x longer blocks the entire stage; cluster utilization drops. **Strategies with trade-offs**: (1) **Salting**: Add random suffix to skewed keys;...
This medium-level Spark/Big Data question appears frequently in data engineering interviews at companies like Accenture, Bitwise. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (join, partition, spark) will help you answer variations of this question confidently.
Break this problem into components. Identify the core trade-offs involved, then walk the interviewer through your reasoning step by step. Demonstrate awareness of edge cases and production considerations - this is what separates good answers from great ones.
Skew occurs when a few keys hold disproportionate data, causing hotspot tasks and stragglers. Why it matters: One task taking 10x longer blocks the entire stage; cluster utilization drops. Strategies with trade-offs: (1) Salting: Add random suffix to skewed keys; distributes load but requires two-phase aggregation (first with salt, then collapse). Cost: 2x shuffle. (2) Broadcast for small dimension: If the skewed side is a small lookup, broadcast avoids shuffle; only works when the skewed table is small. (3) AQE Skew Join (Spark 3.0+): Automatic split of skewed partitions; zero code change, but requires spark.sql.adaptive.enabled=true. (4) Custom partitioning: Range partitioner for known skewed keys; more control, more tuning. Cost implication: Salting adds compute; AQE adds planning overhead; broadcast has driver memory limit. Architectural logic: Prefer AQE first (low effort); add salting when AQE can't fix (e.g., extreme skew). Best practice: Monitor task duration distribution; set spark.sql.adaptive.skewJoin.enabled=true.
Red Flag: 'We just increase partitions'—that doesn't fix skew; skewed keys still land in few partitions. Pro-Move: 'We had a join on user_id where 5% of users had 80% of events; AQE helped but we added salting for the top 100 skewed keys and reduced P99 task time from 45min to 8min.'
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 2 companies. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.