Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why systematic resolution matters**: Ad-hoc fixes don't scale; patterns enable repeatable optimization. **Common challenges & resolutions**: (1) **OOM**—increase executor memory; reduce partition size; avoid collect/broadcast oversized; optimize joins. (2) **Skew**—salting...
Why systematic resolution matters: Ad-hoc fixes don't scale; patterns enable repeatable optimization. Common challenges & resolutions: (1) OOM—increase executor memory; reduce partition size; avoid collect/broadcast oversized; optimize joins. (2) Skew—salting (add random key, redistribute, aggregate); split hot keys; broadcast small side. (3) Slow shuffle—tune spark.sql.shuffle.partitions; broadcast when possible; repartition before wide op. (4) Small files—coalesce/repartition before write; Delta auto-optimize. Scalability trade-offs: Each fix has side effects; salting adds complexity; broadcast has size limits. Cost implications: OOM = retries = 2–3x cost; skew = stragglers = wasted compute. Best practice: Profile with Spark UI; establish baselines; iterate with metrics.
Red Flag: Generic 'we tuned it' without specifics. Pro-Move: 'Spark UI showed skew on user_id; salting reduced runtime from 2hr to 25min.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.