Explain PySpark's Catalyst Optimizer.

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

The Catalyst Optimizer is Spark's rule-based and cost-based query optimizer. It operates on the logical plan (unresolved to resolved to optimized) and physical plan. Key phases: (1) Analysis—resolve attributes and relations; (2) Logical optimization—apply rules (predicate pushdown, constant folding, join reordering); (3) Physical planning—select strategies (broadcast vs shuffle join); (4) Code generation—generate Java bytecode via Tungsten. Example: filter pushdown before join reduces data. Best practices: use DataFrame/Dataset APIs (not RDD) to leverage Catalyst; write queries that enable predicate pushdown; use appropriate join hints when Catalyst chooses suboptimally.