Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Storage levels: MEMORY_ONLY—fast, no spill; use when fits. MEMORY_AND_DISK—spill excess to disk. MEMORY_ONLY_SER—serialized, less memory....
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Storage levels: MEMORY_ONLY—fast, no spill; use when fits. MEMORY_AND_DISK—spill excess to disk. MEMORY_ONLY_SER—serialized, less memory. MEMORY_AND_DISK_SER—serialized + spill. DISK_ONLY—large, rarely reused. Best practice: Default MEMORY_AND_DISK for safety; MEMORY_ONLY for small, hot data.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: MEMORY_ONLY for large. Pro-Move: 'MEMORY_AND_DISK default; profile.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.