Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**getmerge** (HDFS to local): `hadoop fs -getmerge /input/dir /local/merged.txt` **In-HDFS merge** (cat + put): `hadoop fs -cat /input/dir/* | hadoop fs -put - /output/merged` **Spark** (preferred for large data): `spark.read.text("/input/*").coalesce(1).write.text("/output")`...
This medium-level Spark/Big Data question appears frequently in data engineering interviews at companies like Infosys. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (partition, spark) will help you answer variations of this question confidently.
Break this problem into components. Identify the core trade-offs involved, then walk the interviewer through your reasoning step by step. Demonstrate awareness of edge cases and production considerations - this is what separates good answers from great ones.
getmerge (HDFS to local): hadoop fs -getmerge /input/dir /local/merged.txt
In-HDFS merge (cat + put): hadoop fs -cat /input/dir/* | hadoop fs -put - /output/merged
Spark (preferred for large data): spark.read.text("/input/*").coalesce(1).write.text("/output")
Why Avoid Single Huge File: (1) No parallelism for downstream reads. (2) Single block limits map tasks. (3) Memory pressure if read entirely. Merge only when consumer requires single file (e.g., export to external system).
Scalability Trade-offs: getmerge pulls to driver/local—fails for TB. Spark coalesce(1) shuffles all data to one partition; expensive. Prefer multiple files with reasonable size (64–128MB).
Cost Implications: Coalesce(1) on 1TB = full shuffle; use only when necessary. Small-file consolidation (OPTIMIZE in Delta) is different—keeps parallelism.
Pro-Move: 'We use coalesce only for final export sink; internal layers stay partitioned.' Red Flag: Merging 10K small files to one 100GB file—kills downstream parallelism.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.