Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Spark Streaming: `readStream.format('kafka')` or `format('parquet').schema(schema)` for file stream. File mounting: mount S3/ADLS to...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Spark Streaming: readStream.format('kafka') or format('parquet').schema(schema) for file stream. File mounting: mount S3/ADLS to DBFS; use spark.readStream.format('parquet').load('/mnt/path'). Best practice: Checkpoint; handle schema; monitor lag.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: No checkpoint. Pro-Move: 'checkpoint; schema; monitor lag.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.