Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Partitioning**: Input partitioned (e.g., date); repartition to 2–4x cores (e.g., 128–256 for 64 cores). **Format**: Parquet/ORC; columnar, predicate pushdown, compression. **Cluster**: 64–128 cores; executors 4–8 cores, 8–16GB....
Partitioning: Input partitioned (e.g., date); repartition to 2–4x cores (e.g., 128–256 for 64 cores).
Format: Parquet/ORC; columnar, predicate pushdown, compression.
Cluster: 64–128 cores; executors 4–8 cores, 8–16GB. Avoid spill; size memory.
Broadcast: Small lookup tables.
Predicate Pushdown: Filter on partition columns and row groups.
AQE: Enable for coalesce and skew.
Incremental: If possible, process only new/changed data.
Spot: Use spot for batch; 60–70% savings.
Scalability Trade-offs: 1TB / 128MB ≈ 8K partitions; default 200 shuffle partitions may underutilize. Tune spark.sql.files.maxPartitionBytes.
Cost Implications: 1TB at 10 min = ~100 core-hours. Spot + right-size = $5–15 for run.
Pro-Move: 'We partition by date; process only last 7 days incrementally; 90% less data.' Red Flag: Reading 1TB without partition pruning—full scan.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.