What factors determine the optimal number of partitions for a large file?

Optimal partition count factors: (1) Data size—target 100MB–1GB per partition to avoid small files. (2) Query patterns—partition by filter columns (date, region). (3) Parallelism—partitions = parallelism in Spark; balance with overhead. (4) File count—too many partitions = small files, Namenode pressure (HDFS). (5) Skew—avoid high-cardinality columns that cause imbalance. Rule of thumb: num_partitions ≈ sqrt(data_size_GB) or match core count for single job. In Spark: repartition by column for even distribution; coalesce to reduce. Best practice: Align partitions with downstream consumers; monitor partition pruning in query plans. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.