How do you handle data skewness in Spark?

Skew occurs when a few keys hold disproportionate data, causing hotspot tasks and stragglers. Why it matters: One task taking 10x longer blocks the entire stage; cluster utilization drops. Strategies with trade-offs: (1) Salting: Add random suffix to skewed keys; distributes load but requires two-phase aggregation (first with salt, then collapse). Cost: 2x shuffle. (2) Broadcast for small dimension: If the skewed side is a small lookup, broadcast avoids shuffle; only works when the skewed table is small. (3) AQE Skew Join (Spark 3.0+): Automatic split of skewed partitions; zero code change, but requires spark.sql.adaptive.enabled=true. (4) Custom partitioning: Range partitioner for known skewed keys; more control, more tuning. Cost implication: Salting adds compute; AQE adds planning overhead; broadcast has driver memory limit. Architectural logic: Prefer AQE first (low effort); add salting when AQE can't fix (e.g., extreme skew). Best practice: Monitor task duration distribution; set spark.sql.adaptive.skewJoin.enabled=true.