Explain Spark transformations (lazy evaluation, wide vs narrow).

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Spark transformations are lazy—they build a DAG and execute only when an action triggers. Narrow transformations (map, filter) don't require data movement; each partition is processed independently. Wide transformations (groupBy, join, distinct) require shuffle—data is redistributed across partitions. Narrow: df.filter(col('x')>0). Wide: df.groupBy('key').sum(). Best practices: minimize wide transformations; use coalesce to reduce partitions after filter (avoids unnecessary shuffle); prefer reduceByKey over groupByKey; use broadcast joins when one side is small. Lazy evaluation enables optimization (e.g., predicate pushdown) and pipelining of narrow operations.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.