Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Process large log file for top 10 users: Use Spark: `spark.read.text(path).select(split(col('value'),'...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Process large log file for top 10 users: Use Spark: spark.read.text(path).select(split(col('value'),' ').getItem(0).alias('user')).groupBy('user').count().orderBy(desc('count')).limit(10). For streaming: readStream with watermark. Optimize: partition by date; use reduceByKey equivalent; avoid collect. Best practice: Incremental processing; checkpoint for streaming.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: collect for top N. Pro-Move: 'limit; reduceByKey; streaming watermark.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.