Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**cache()**: Equivalent to `persist(MEMORY_AND_DISK)`. Stores partitions in memory; spills to disk if memory is insufficient. **persist(storage_level)**: Explicit control over storage: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY....
Pro-Move: Tie storage level to cluster sizing and cost. Red Flag: Caching and never unpersisting—memory leak and wasted spend.
This medium-level Spark/Big Data question appears frequently in data engineering interviews at companies like Accenture, Coforge, Freecharge, and 2 others. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (partition, spark) will help you answer variations of this question confidently.
Break this problem into components. Identify the core trade-offs involved, then walk the interviewer through your reasoning step by step. Demonstrate awareness of edge cases and production considerations - this is what separates good answers from great ones.
cache(): Equivalent to persist(MEMORY_AND_DISK). Stores partitions in memory; spills to disk if memory is insufficient.
persist(storage_level): Explicit control over storage: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY.
Architectural Logic (Why It Matters): Caching trades memory/disk for recomputation cost. The right choice depends on reuse count, data size, serialization overhead, and cluster resources.
Scalability & Cost Trade-offs:
Cost Implications: Caching a 500GB DataFrame in MEMORY_ONLY on 100 executors with 8GB each = eviction thrashing. Use MEMORY_AND_DISK or MEMORY_ONLY_SER. Always unpersist() when done to free resources and avoid unnecessary cluster cost.
Pro-Move: Tie storage level to cluster sizing and cost. Red Flag: Caching and never unpersisting—memory leak and wasted spend.
Practice the 33 most asked data engineering questions at Accenture. Covers SQL, Spark/Big Data, Behavioral and more.
8 min read →Senior Spark interviews at Amazon, Databricks, and Meta focus on performance tuning, not API syntax. Master these 15 questions to prove you've run Spark at scale.
20 min read →Compare your response with a stronger, structured example and identify missing concepts.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 5 companies. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.