Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Spark: partition by key (date, id) for pruning and join locality. Cache: cache() for repeated use. Parallelism: spark.default.parallelism = 2–4 * cores; repartition for shuffle. Tuning: executor memory, cores, shuffle partitions. AQE for adaptive optimization. Monitor Spark UI...
Spark: partition by key (date, id) for pruning and join locality. Cache: cache() for repeated use. Parallelism: spark.default.parallelism = 2–4 * cores; repartition for shuffle. Tuning: executor memory, cores, shuffle partitions. AQE for adaptive optimization. Monitor Spark UI for skew and GC. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.
Red Flag: Arbitrary partition counts. Pro-Move: 'Partitions = size/150MB; we monitored Spark UI for skew, adjusted when P99 > 2× median.'
According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.