Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why this matters at scale**: Failures and misconfig cost time and money; optimization compounds across thousands of jobs. **Failures & resolutions**: Executor OOM—increase memory, reduce partition size, or optimize (avoid broad joins). Driver OOM—avoid collect(); use...
Why this matters at scale: Failures and misconfig cost time and money; optimization compounds across thousands of jobs. Failures & resolutions: Executor OOM—increase memory, reduce partition size, or optimize (avoid broad joins). Driver OOM—avoid collect(); use checkpointing for long lineage. Stragglers—speculative execution, salting for skew. Optimization: Partition pruning, predicate pushdown, broadcast joins, AQE. Resource utilization: Right-size executors (4–8 cores, 8–16GB); dynamic allocation for variable load; monitor Spark UI. Scalability trade-offs: Over-provisioning = cost; under-provisioning = OOM and retries. Cost implications: Spot instances for batch; reserved for critical; tune spark.sql.shuffle.partitions to reduce shuffle cost.
Red Flag: Blindly increasing memory without profiling. Pro-Move: 'We profiled with Spark UI, found skew in join—salting dropped runtime 70%.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.