Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why Spark fundamentals matter**: Understanding abstractions and execution model directly impacts performance tuning and cost. **Core abstractions**: RDD (immutable, partitioned, lineage); DataFrame/Dataset (structured, Catalyst-optimized). **Lazy evaluation**: Transformations...
Why Spark fundamentals matter: Understanding abstractions and execution model directly impacts performance tuning and cost. Core abstractions: RDD (immutable, partitioned, lineage); DataFrame/Dataset (structured, Catalyst-optimized). Lazy evaluation: Transformations build DAG; actions trigger execution—enables predicate pushdown, pipelining. APIs: Spark SQL, Structured Streaming, MLlib, GraphX. Scalability trade-offs: In-memory speeds iteration but OOM risk; disk spill when memory exhausted. DataFrame > RDD for optimization surface. Cost implications: Wrong partition count = wasted executors or small-file sprawl; AQE helps but initial config matters. Best practice: Prefer DataFrame; tune partitions; use columnar formats for I/O.
Red Flag: Calling RDD and DataFrame interchangeable. Pro-Move: 'DataFrame leverages Catalyst; RDD for custom partitioning logic only.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.