Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Property**: `spark.sql.shuffle.partitions`. Default 200. Controls output partitions after shuffle (groupBy, join). **Set**: `spark.conf.set("spark.sql.shuffle.partitions", 400)` or `--conf spark.sql.shuffle.partitions=400`. **Tuning**: 2–4x core count for small data. For...
Property: spark.sql.shuffle.partitions. Default 200. Controls output partitions after shuffle (groupBy, join).
Set: spark.conf.set("spark.sql.shuffle.partitions", 400) or --conf spark.sql.shuffle.partitions=400.
Tuning: 2–4x core count for small data. For large data, higher (e.g., 400–800). Too low = large partitions, spill. Too high = overhead.
AQE: Can coalesce at runtime. Set initial high; AQE reduces.
Cost Implications: Right count = balanced tasks. Wrong = skew or overhead.
Pro-Move: 'Default 200; we set 2x cores, AQE coalesces when small.' Red Flag: 200 on 1TB join—200 huge partitions; 2000 on 10GB—overhead.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.