What is Adaptive Query Execution (AQE) in Spark 3.x, and how does it improve performance?

AQE re-optimizes at runtime using actual statistics at stage boundaries, addressing the planning-time blind spot (e.g., wrong size estimates, skew). Three features: (1) Coalesce shuffle partitions—merges small partitions after shuffle to reduce task overhead; avoids 10K tiny tasks. (2) Switch join strategy—if one side is smaller than expected, converts sort-merge to broadcast; avoids unnecessary shuffle. (3) Skew join—splits skewed partitions into smaller tasks; eliminates stragglers. Why it matters: Static plans assume uniform data; real data is skewed and irregular. Scalability: AQE adds planning overhead (~100ms per stage); payoff is large for skewed or unpredictable workloads. Cost implication: Can reduce job runtime 20–50% without manual tuning; reduces need for over-provisioning. Enable: spark.sql.adaptive.enabled=true. Best practice: Enable AQE by default on Spark 3.x; combine with spark.sql.adaptive.coalescePartitions.enabled for shuffle coalescing.