Explain how Spark groups transformations into stages. What causes a stage boundary?

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Spark groups transformations into stages. A stage boundary occurs at a shuffle dependency—when data must be redistributed (e.g., groupBy, join, repartition). Transformations before the shuffle execute in one stage; transformations requiring the shuffle form the next. Narrow dependencies allow pipelining within a stage. Example: filter, map, groupBy, agg: stages 1 (filter+map), stage 2 (groupBy+agg). Best practices: minimize stages; push filters before shuffles; use broadcast to eliminate shuffle when one side is small.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.