How do you optimize Spark jobs for performance?

Optimization is a hierarchy: (1) Reduce data scanned—partition pruning, predicate pushdown, column pruning; biggest lever. (2) Reduce shuffle—broadcast small tables, avoid unnecessary repartitions, co-locate joins. (3) Right-size parallelism—spark.sql.shuffle.partitions ~2–4× cores; too many = task overhead; too few = underutilization. (4) Avoid serialization hot paths—use built-in functions over UDFs; UDFs break Catalyst and force row-by-row Python or slower Java. (5) Persistence strategy—cache only reused DFs; unpersist to free memory. (6) Skew—use AQE or salting. Cost implication: A 10% reduction in shuffle can cut runtime 20–30% on I/O-bound jobs. Scalability: As data grows, partition count and memory per executor become critical; monitor GC and spill metrics. Architectural logic: Optimize in order—data reduction first, then shuffle, then compute. Best practice: Use Spark UI to identify bottlenecks; enable AQE; profile before tuning.