Coforge Data Engineer Interview Questions (2026)

**Narrow transformations**: Each input partition maps to at most one output partition. No shuffle. Examples: map, filter, flatMap, mapPartitions. **Wide transformations**: Require data from multiple input partitions to produce one output partition. Trigger shuffle. Examples: groupByKey, reduceByKey, join, distinct, repartition. **Architectural Logic (Why This Matters)**: Spark pipelines narrow transformations and executes them in a single stage. Wide transformations force a stage boundary—all prior work is materialized (shuffle write), then a new stage reads (shuffle read). Stage count and shuffle volume drive job latency and cost. **Scalability Trade-offs**: - Minimize wide transforms: each adds network I/O and potential skew. - Order matters: filter (narrow) before join (wide) to reduce shuffle size. - broadcast() converts a join into a narrow op—use when one side is small. **Cost Implications**: A pipeline with 5 wide transforms = 5 shuffles. Reordering to 2 wide transforms can cut runtime by 50%+. Partition pruning and predicate pushdown (narrow) reduce data before expensive wide ops. **Examples**: ```python # Narrow: pipelineable within same stage df.filter(col("age") > 18).select("id", "amount") # Wide: triggers shuffle; stage boundary df.groupBy("dept").agg(sum("salary")) ```