Cache() vs Persist(): Explain the difference and use cases for caching and persisting data in Spark with memory levels.

Why this matters: Wrong storage level = OOM or unnecessary disk I/O. cache() = persist(StorageLevel.MEMORY_AND_DISK)—default for quick reuse. persist() accepts levels: MEMORY_ONLY (fast, evicted if OOM), MEMORY_AND_DISK (spill to disk), MEMORY_ONLY_SER (serialized, less memory), MEMORY_AND_DISK_SER, DISK_ONLY. Use cases: MEMORY_ONLY for hot small DataFrames; MEMORY_AND_DISK when data may not fit; serialized for memory savings. Scalability trade-offs: MEMORY_ONLY eviction = recompute; MEMORY_AND_DISK = disk I/O on eviction. Cost implications: Unpersist when done—holding memory blocks other tasks. Checkpoint for lineage truncation when DAG is deep. Best practice: Profile; prefer built-in operators over manual cache when possible.