Explain Spark's execution process – Job/Stage/Task creation.

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Spark execution: (1) Action triggers job; (2) DAG Scheduler splits the DAG into stages at wide transformation boundaries (shuffle dependency); (3) Each stage has tasks (one per partition); (4) Task Scheduler launches tasks on executors. Stage boundary occurs when a shuffle is required—e.g., before a reduceByKey or join. Stages without shuffle dependencies can be pipelined. Example: filter, map, groupBy, collect creates 2 stages (filter+map in stage 1, groupBy in stage 2). Best practices: minimize stages; reduce shuffle via broadcast; use appropriate partition count to balance parallelism and overhead.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.