Explain wide vs. narrow transformations and how they drive shuffle cost, failure domains, and pipeline design. When would you intentionally add a wide transformation, and how do you minimize its impact?

Section 1 — The Context (The 'Why')
Wide transformations force full data shuffles across the cluster; narrow transformations stay partition-local. The cost of shuffle dominates Spark job runtime at scale.

Section 2 — The Diagram

[Narrow: map, filter] --> [RDD] --> [Wide: join, groupBy] --> [Shuffle]

Section 3 — Component Logic
Narrow transformations (map, filter) do not require data movement. Wide transformations (join, groupByKey) trigger shuffle—data redistributed by key. Partitioning strategies before wide ops reduce shuffle. Data skew mitigation via salting. Backpressure handling in streaming adapts batch size. Exactly-once semantics require deterministic partitioning. Why: narrow ops are free; wide ops expensive—minimize, use broadcast when possible.

Section 4 — The Trade-offs (The 'Senior' part)

Section 5 — Pro-Tip
Pro-Move: 'Broadcast for dim tables cut shuffle from 200GB to 40GB.' Red Flag: groupBy without understanding shuffle.

Supplemental (Senior Context): In production, monitor partition skew, consumer lag, and merge duration. Use correlation IDs for traceability across pipeline stages. Schema evolution: prefer additive changes only; use Schema Registry for streaming to enforce compatibility. Consider data contract tests in CI to catch breaking changes early. Budget 10-20% overhead for replication, checkpoint storage, and DLQ. Data quality gates at each layer prevent bad data propagation. Right-size resources: profile before scaling; over-provisioning wastes budget. Document runbooks for common failures: broker restart, consumer rebalance, sink timeout. Establish SLOs per stage: ingest latency, transform duration, serve freshness. Review partition key choice: avoid high-cardinality keys that cause explosion; use composite keys (date, tenant) for balanced distribution. Test failure injection: kill executors, broker, sink to validate recovery. Optimize for the common case: most queries filter by date. Cold start mitigation: pre-warm connections, cache dimension lookups. Alert on lag exceeding 1hr, error rate above 1%. Cost optimization: lifecycle policies, spot instances, partition pruning. Lineage tracking enables impact analysis. Idempotency keys for replay. Backpressure handling prevents slow consumers from blocking producers. Fan-out patterns allow multiple consumers without re-processing. Exactly-once semantics require replayable source and idempotent sink. Data skew mitigation via salting for high-cardinality joins. Partitioning strategies must align with query patterns for pruning. CAP trade-off: AP for ingest and transform; CP for serve when BI needs accuracy. Blast radius bounded by partition and consumer group. Measure and iterate: latency percentiles, cost per record, error rate. Principal engineer tip: quantify before and after optimizations. Red flag: describing architecture without trade-offs. Glue versus EMR: Glue for bursty sub-2hr jobs; EMR for sustained 8hr+ saving 60%. MSK for Kafka; S3 for lake storage. Self-heal: orchestration retries; idempotent sinks ensure consistency. If primary fails, downstream goes stale but no data loss with replay. Design for operability: runbooks, dashboards, alerts. Avoid tight coupling between stages. Incremental processing reduces compute versus full refresh. Watermark-based deduplication enables idempotency. Partition evolution: add new partitions without rewriting. Retention policies balance cost and compliance. Test at scale: use production-size samples for validation. Always document trade-offs.