Freight Tiger Data Engineer Interview Questions (2026)

**Architecture**: Job = one action; Stage = boundary at shuffle; Task = unit per partition. Stages enable pipelining of narrow transformations (filter, map) across partitions without network I/O; shuffles force stage boundaries and dominate cost. **Why it matters for sizing**: Cluster parallelism is bounded by min(#tasks, #cores). Over-partitioning increases tasks and overhead (scheduler, task launch); under-partitioning underutilizes clusters. Rule of thumb: 2–4x cores per executor for CPU-bound; more for I/O-bound. **Scalability trade-offs**: Fewer stages = less shuffle = lower cost but less opportunity for optimization (e.g., predicate pushdown per stage). More stages from excessive shuffles (e.g., multiple groupBys) increase network and disk I/O. Use coalesce to reduce tasks after wide transformations when downstream logic doesn't need parallelism. **Cost implications**: Shuffles drive S3/HDFS reads and network; they scale with O(n) in data size. Reducing shuffle data (select only needed columns, avoid skew) directly reduces cost. Use Spark UI: Jobs → Stages → Tasks to correlate stage duration with shuffle read/write. Example: df.filter().groupBy().count() → 1 job, 2 stages (pre-shuffle, post-shuffle), N tasks per stage. Tuning: repartition before shuffle for skew mitigation; increase partitions only when parallelism is the bottleneck.