Walk through the three AQE features in Spark 3.x (coalesce, join switch, skew join)—how they operate at shuffle boundaries, which configs enable them, and what happens when AQE cannot help.

AQE in Spark 3.x performs runtime reoptimization at shuffle boundaries. Three features: (1) Coalesce shuffle partitions (spark.sql.adaptive.coalescePartitions.enabled): post-shuffle, merge undersized partitions into fewer tasks—reduces scheduler overhead and small-task waste. (2) Join strategy switch: if runtime stats show one side small, convert sort-merge to broadcast—eliminates shuffle for that side. (3) Skew join: detect oversized partitions from shuffle stats; split and process in parallel. Config: spark.sql.adaptive.enabled=true; coalesce/skew/join switches each have enable flags. AQE cannot help when: no shuffle (narrow-only stage); skew in sort/aggregate before shuffle; stats are wrong and hint overrides. Scalability: AQE reduces manual tuning; initial shuffle partitions (e.g., 200) matter since coalesce merges down. Cost: minor runtime overhead; large savings from fewer wasted tasks.