Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Advantage**: Avoid recomputation. Same DF used multiple times = one compute, many reads. Critical for iterative algorithms (ML), multi-action pipelines. **When**: (1) DF reused 2+ times. (2) Iterative (e.g., loop with same lookup). (3) Small dimension joined repeatedly....
Advantage: Avoid recomputation. Same DF used multiple times = one compute, many reads. Critical for iterative algorithms (ML), multi-action pipelines.
When: (1) DF reused 2+ times. (2) Iterative (e.g., loop with same lookup). (3) Small dimension joined repeatedly.
Storage Levels: MEMORY_ONLY (fast, evictable), MEMORY_AND_DISK (spill), MEMORY_ONLY_SER (smaller). Cache = MEMORY_AND_DISK.
Why Care: Recompute of 1TB = hours. Cache = seconds for second read.
Scalability Trade-offs: Cache consumes memory; evicts other data. Unpersist when done. Over-caching = thrashing.
Cost Implications: Cache when beneficial = 50–90% cost reduction for multi-use. Cache everything = OOM and waste.
Pro-Move: 'Cache only after filter; we cache 10MB dim, not 100GB fact.' Red Flag: Caching and never unpersisting—memory leak across jobs.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.