Explain the concept of checkpointing in Spark and why it is important.

Checkpointing breaks RDD lineage by materializing to storage, trading storage I/O for DAG truncation. Why it exists: Long lineage (e.g., 1000+ stages in iterative algorithms like PageRank or ML) causes stack overflow in the driver and makes fault recovery expensive—Spark would replay the entire DAG. Scalability trade-off: Checkpoint is a full write; at scale, this is costly (network, disk). For batch, prefer persist/cache when lineage is < ~50 stages. Cost implication: Checkpointing to S3 adds egress and request costs; HDFS checkpoint is cheaper but requires cluster storage. Streaming: Checkpoint is mandatory for exactly-once semantics—it stores offsets and state; without it, restart loses position. Architectural logic: Use checkpoint when lineage length or recovery time is the bottleneck; otherwise prefer lazy evaluation. Best practice: checkpointDirectory on durable storage (S3/HDFS); for streaming, always use checkpoint; for batch iterative jobs, checkpoint every N iterations.