Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Essential Properties**: `--master` (yarn, local, k8s), `--deploy-mode` (client/cluster), `--driver-memory`, `--executor-memory`, `--num-executors`, `--executor-cores`. `--conf` for fine-grained: `spark.sql.shuffle.partitions`, `spark.dynamicAllocation.enabled`. **Why They...
Essential Properties: --master (yarn, local, k8s), --deploy-mode (client/cluster), --driver-memory, --executor-memory, --num-executors, --executor-cores. --conf for fine-grained: spark.sql.shuffle.partitions, spark.dynamicAllocation.enabled.
Why They Matter: Wrong memory = OOM or waste. Too few executors = underutilization. Too many = overhead. shuffle.partitions default (200) often wrong for large data.
Scalability Trade-offs: cluster deploy mode runs driver in cluster—resilient to client disconnect. Dynamic allocation scales executors by backlog; fixed num-executors simpler but less elastic.
Cost Implications: Over-provisioning wastes 20–40%. Start with --executor-memory 4g, --num-executors = (data_partitions / cores_per_executor). Use --packages for dependencies; --py-files for Python libs.
Pro-Move: 'We use spark-submit --config file.conf; version-controlled tuning.' Red Flag: Default shuffle.partitions=200 on 10TB join—massive overhead.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.