Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Scala Spark: df.dropDuplicates().na.fill(0).filter(col("amount").isNotNull). For schema: df.filter(col("col").cast("int").isNotNull) or use schemaOf and validate. With Dataset: ds.dropDuplicates("id").na.fill(Map("col" -> 0)). Schema validation: expectStructType or try/catch on...
Scala Spark: df.dropDuplicates().na.fill(0).filter(col("amount").isNotNull). For schema: df.filter(col("col").cast("int").isNotNull) or use schemaOf and validate. With Dataset: ds.dropDuplicates("id").na.fill(Map("col" -> 0)). Schema validation: expectStructType or try/catch on read. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.
Red Flag: Changing schema in prod without rollback testing. Pro-Move: 'We used Delta mergeSchema on 1% sample first; full backfill in maintenance window.'
According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.