Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Fill nulls in PySpark: (1) `fillna(value)` or `fillna(dict)` for specific columns: `df.fillna(0)` or...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Fill nulls in PySpark: (1) fillna(value) or fillna(dict) for specific columns: df.fillna(0) or df.fillna({'col1':0,'col2':'unknown'}). (2) na.fill(). (3) when/otherwise: df.withColumn('col', when(col('col').isNull(), lit(0)).otherwise(col('col'))). For forward/back fill: Use window functions. Best practice: Document fill strategy; avoid masking data quality issues; use appropriate types (e.g., 0 for numeric, 'unknown' for string).
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Masking data quality. Pro-Move: 'Document strategy; validate; avoid silent fill.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.