Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why Heap Matters:** Executor runs tasks; each task uses JVM heap. OOM when task + overhead exceeds spark.executor.memory. memoryOverhead (Python, off-heap) is separate—both count toward container limit. **Tuning:** Increase executor.memory for large shuffles. Rule: (cluster...
Why Heap Matters: Executor runs tasks; each task uses JVM heap. OOM when task + overhead exceeds spark.executor.memory. memoryOverhead (Python, off-heap) is separate—both count toward container limit.
Tuning: Increase executor.memory for large shuffles. Rule: (cluster memory / num executors) - overhead. Also: more partitions (smaller per-task), avoid collect(), use broadcast for small tables, cache selectively.
Cost: Larger executors = fewer executors for fixed cluster. Too few = underutilization; too many = overhead. Sweet spot: 4–8 cores, 8–16GB per executor. Monitor Spark UI for spill and GC.
spark.executor.memory=8g
spark.executor.memoryOverhead=2g
Red Flag: Blaming OOM without checking spill and partition size. Pro-Move: 'We increased partitions to 2x cores and reduced skew—OOMs dropped; memoryOverhead 20% for Python.'
According to DataEngPrep.tech, this is one of the most frequently asked Python/Coding interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.