Question 1

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Accepted Answer

**Narrow transformations**: Each input partition maps to at most one output partition. No shuffle. Examples: map, filter, flatMap, mapPartitions.

**Wide transformations**: Require data from multiple input partitions to produce one output partition. Trigger shuffle. Examples: groupByKey, reduceByKey, join, distinct, repartition.

**Architectural Logic (Why This Matters)**: Spark pipelines narrow transformations and executes them in a single stage....

Question 2

Explain the difference between Spark's map() and flatMap() transformations.

Accepted Answer

map(): 1-to-1; each input element yields one output. flatMap(): 1-to-many or 1-to-0; function returns iterable, results flattened. Example: map splits string → one string; flatMap splits → multiple words. Why it matters: map preserves partition count; flatMap can change it (e.g., explode JSON array → more rows). Use flatMap for: tokenization, exploding arrays, filtering (return empty list). Performance: flatMap that expands heavily can cause skew; monitor partition sizes....

Question 3

Explain the concept of Broadcast Join in Spark. When should it be used?

Accepted Answer

Mechanism: Small table sent to all executors; join happens locally, no shuffle. Triggered by broadcast() hint or spark.sql.autoBroadcastJoinThreshold (default 10MB). Why: Shuffle of large table is expensive; broadcasting small table avoids it. When: One side fits in executor memory (~broadcast threshold). Trade-off: Too large = driver/executor OOM; too small threshold = unnecessary shuffles. Cost: Broadcast data replicated per executor; acceptable for MB-scale....

Question 4

When and how do you use Broadcast Join in Spark?

Accepted Answer

Broadcast join sends the small table to every executor; join happens locally without shuffle. **Trigger**: broadcast(df) hint or spark.sql.autoBroadcastJoinThreshold (default 10MB). **Why use it**: Avoids shuffling the large table—the dominant cost in sort-merge join. **When**: One table fits in executor memory (typically < 100MB after serialization). **Scalability trade-off**: Driver fetches and distributes; oversized broadcast causes driver OOM....

Question 5

How do you compare the time investment and value of a task?

Accepted Answer

**Situation**: Faced competing demands—multiple pipelines, stakeholders, deadlines. **Task**: Deliver impact while maintaining quality and preventing burnout. **Action**: (1) Prioritized by business impact and SLA risk. (2) Used ROI (value/time); WIP limits; timeboxing. (3) Communicated trade-offs—'Adding X pushes Y by N days.' (4) Maintained backlog with tech-debt capacity. **Result**: Shipped on time; zero incidents; stakeholder alignment on deferrals....

Question 6

What is offset management in Kafka?

Accepted Answer

**Offset**: Position in partition log. Consumer commits offset to track progress. Stored in __consumer_offsets (Kafka) or external (Spark checkpoint).

**Modes**: Auto-commit (periodic) vs. manual (commitSync/commitAsync after process). Manual required for exactly-once.

**Why It Matters**: Lost offset = reprocess or skip. Exactly-once needs atomic process+commit.

**Scalability Trade-offs**: Rebalance invalidation; consumer must handle....

Question 7

What metrics do you use to determine whether a Spark job is going well or not?

Accepted Answer

**Good Signs**: Even task duration (no skew). Low GC time. No spill. High executor compute time vs. total. Input bytes processed. No task failures.

**Bad Signs**: Skewed task duration (one 10x longer). High GC. Spill (memory/disk). Low compute ratio. Task failures. Growing backlog.

**Spark UI Metrics**: Stage duration, shuffle read/write, GC time, spill. Task duration histogram. DAG.

**SLA Metrics**: End-to-end duration, success rate....

Delivery Hero Spark & Big Data Interview Questions

Companies asking these questions

All 7 Questions

More Interview Prep Guides

Get All Answers in PDF Format

Delivery Hero Spark & Big Data Interview Questions

Companies asking these questions

All 7 Questions

More Interview Prep Guides

Get All Answers in PDF Format