Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Drop duplicates: `df.dropDuplicates()` or `df.dropDuplicates(['col1','col2'])`. Keeps first occurrence. For deterministic:...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Drop duplicates: df.dropDuplicates() or df.dropDuplicates(['col1','col2']). Keeps first occurrence. For deterministic: df.orderBy('ts').dropDuplicates(['key']). Best practice: Define key; handle nulls; consider window for 'keep latest'.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: dropDuplicates non-deterministic. Pro-Move: 'orderBy before; window for latest.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.