What is Spark's Catalyst Optimizer? Explain its stages.

Catalyst is Spark SQL's extensible query optimizer—it converts logical plans to physical plans with rule-based and cost-based optimizations. Stages: (1) Analysis—resolve table/column references using Catalog; bind types. (2) Logical optimization—apply rules: predicate pushdown (filter at scan), projection pruning (drop unused columns), constant folding, join reordering. (3) Physical planning—generate candidate plans (e.g., broadcast vs. sort-merge join); select via cost model. (4) Codegen—generate Java bytecode (Tungsten) for tight loops. Why it matters: Same DataFrame code can produce different physical plans based on data; optimizer adapts. Scalability: Optimizer runs on driver; complex queries with many joins can have longer planning time. Cost implication: Predicate pushdown can reduce scan by orders of magnitude; bad plans (e.g., Cartesian join) can blow up cost. Best practice: Use DataFrame API to get Catalyst benefits; avoid UDFs that break optimization.