Python for Data Engineers Interview Questions (2026)

**Repartition(n)**: Performs a full shuffle to redistribute data across exactly `n` partitions. Can increase or decrease partition count. Uses hash partitioning by default—all rows are exchanged across the network. **Coalesce(n)**: Merges existing partitions into fewer partitions without a full shuffle. Only decreases partition count. Data is combined locally where possible; minimal network transfer. **Why It Matters (Architectural Logic)**: Shuffles are the most expensive operations in Spark—they drive network I/O, serialization, and memory pressure. Choosing the wrong method can 2–3x your job runtime and cost. **Scalability Trade-offs**: - **repartition**: Use when increasing parallelism (e.g., from 10 → 200 partitions before a heavy aggregation) or when data is severely skewed and you need even redistribution. Cost: full shuffle. - **coalesce**: Use when reducing partitions before writes to avoid small files (e.g., coalesce(1) or coalesce(num_output_files)). Cost: minimal; risk of skew if coalescing too aggressively. **Cost Implications**: On a 1TB dataset, repartition(1) would shuffle ~1TB; coalesce(1) shuffles only the overflow. Prefer coalesce when decreasing partitions to control write file count and cloud storage costs. **Example**: ```python # Increase parallelism before expensive wide transform (full shuffle) df_repartitioned = df.repartition(200) # Reduce before write to avoid small-files problem (minimal shuffle) df_coalesced = df.coalesce(10) ```