Explain how Adaptive Query Execution changes the economics of Spark tuning. What problems does it solve at runtime, and when might you still need manual intervention (e.g., salting, broadcast hints)?

AQE runs at shuffle boundaries: recalculates partition counts, join strategies, and skew using runtime statistics. Features: (1) Coalesce shuffle partitions—merge small partitions post-shuffle, fewer tasks. (2) Switch sort-merge to broadcast when stats show small side. (3) Skew join—split oversized partitions. Why it matters economically: reduces manual tuning—fewer hours on config, fewer job failures from bad stats. Scalability: AQE adapts to actual data; handles cases where compile-time stats are stale or missing. When manual work still needed: broadcast hints when optimizer underestimates; salting when skew is in non-shuffle operations (e.g., sort); explicit repartition for downstream sink expectations. Cost: AQE adds minor overhead; benefit from fewer wasted tasks and better resource use. Enable by default on Spark 3.x/Databricks; set initial shuffle partitions (e.g., 200) since AQE coalesces.