Reviewed by Aditya Kumar · Last reviewed 2026-03-24
REPARTITION: full shuffle; creates N partitions; use when changing partition count or key. COALESCE: reduces partitions without full shuffle; combines partitions; use before writes to reduce output files. repartition(200) vs coalesce(50): repartition spreads data; coalesce...
REPARTITION: full shuffle; creates N partitions; use when changing partition count or key. COALESCE: reduces partitions without full shuffle; combines partitions; use before writes to reduce output files. repartition(200) vs coalesce(50): repartition spreads data; coalesce merges. Use coalesce when reducing; repartition when increasing or changing key. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.
Red Flag: Generic textbook answers. Pro-Move: 'At scale we measured X, implemented Y, achieved Z%—validated and iterated.'
According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.