Explain repartition vs. coalesce. Which one would you use to reduce shuffle operations?

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

repartition(n) creates n partitions via full shuffle; coalesce(n) merges partitions without full shuffle (narrow). Use coalesce to reduce partitions (e.g., after filter)—avoids shuffle. Use repartition to increase partitions or change partition key. To reduce shuffle: coalesce. Example: df.filter(...).coalesce(10) to shrink after filter. Repartition triggers shuffle. Best practice: coalesce to reduce (e.g., before writing fewer files); repartition when need more parallelism or different partitioning.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.