Explain the difference between partition count and query performance in Spark.

Partition count directly drives parallelism: N partitions → up to N concurrent tasks. Why it matters: Too few partitions underutilize the cluster (e.g., 4 partitions on 64 cores); too many cause scheduler overhead (10K tasks with 100ms overhead each = 16+ mins wasted). Scalability trade-offs: Sweet spot ~2–4× core count; partition size 128–200MB for optimal I/O. At 1TB dataset: 5K–8K partitions is reasonable; 50K partitions = small-file problem, slow metadata, merge overhead. Cost implications: Over-partitioning increases shuffle writes, driver memory for task metadata, and S3 LIST operations. Under-partitioning causes OOM and stragglers. Repartition post-shuffle when downstream needs different granularity; coalesce before write to avoid 10K small files.