Explain a scenario-based question on Spark optimization and how you would troubleshoot performance issues.

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Troubleshooting Spark performance: (1) Identify bottleneck—check Spark UI for long-running stages, skew, or spill; (2) Skew—use salting for hot keys, increase partitions; (3) Spill—increase executor memory or reduce partition size; (4) Slow shuffles—use broadcast for small joins, coalesce after filter; (5) Small files—use repartition/coalesce before write, enable auto-optimize in Delta; (6) Catalyst—ensure predicate pushdown, avoid UDFs when possible. Tools: Spark UI (stages, tasks, shuffle read/write), Spark listener, logs. Example: skewed join fix—add salt: df.withColumn('salt', (rand()*10).cast('int')); then group by (key, salt) and aggregate.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.