Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Remove duplicates in PySpark: (1) `df.dropDuplicates()`—all columns. (2) `df.dropDuplicates(['col1','col2'])`—subset. (3)...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Remove duplicates in PySpark: (1) df.dropDuplicates()—all columns. (2) df.dropDuplicates(['col1','col2'])—subset. (3) df.distinct()—full row. (4) For deterministic 'first' row: df.dropDuplicates(['key']).orderBy('timestamp', ascending=False) with window. Best practice: Define business key; handle nulls in key; use dropDuplicates before expensive joins; consider watermark for streaming.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: dropDuplicates without key. Pro-Move: 'Define business key; window for latest.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.