Question 1

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Accepted Answer

**Narrow transformations**: Each input partition maps to at most one output partition. No shuffle. Examples: map, filter, flatMap, mapPartitions.

**Wide transformations**: Require data from multiple input partitions to produce one output partition. Trigger shuffle. Examples: groupByKey, reduceByKey, join, distinct, repartition.

**Architectural Logic (Why This Matters)**: Spark pipelines narrow transformations and executes them in a single stage....

Question 2

Explain the difference between Spark's map() and flatMap() transformations.

Accepted Answer

map(): 1-to-1; each input element yields one output. flatMap(): 1-to-many or 1-to-0; function returns iterable, results flattened. Example: map splits string → one string; flatMap splits → multiple words. Why it matters: map preserves partition count; flatMap can change it (e.g., explode JSON array → more rows). Use flatMap for: tokenization, exploding arrays, filtering (return empty list). Performance: flatMap that expands heavily can cause skew; monitor partition sizes....

Question 3

Explain the concept of Broadcast Join in Spark. When should it be used?

Accepted Answer

Mechanism: Small table sent to all executors; join happens locally, no shuffle. Triggered by broadcast() hint or spark.sql.autoBroadcastJoinThreshold (default 10MB). Why: Shuffle of large table is expensive; broadcasting small table avoids it. When: One side fits in executor memory (~broadcast threshold). Trade-off: Too large = driver/executor OOM; too small threshold = unnecessary shuffles. Cost: Broadcast data replicated per executor; acceptable for MB-scale....

Question 4

What is the difference between a list and a tuple in Python?

Accepted Answer

List: Mutable, []; tuple: immutable, (). Why it matters: Mutability drives use—lists for collections that change; tuples for fixed data, dict keys (hashable), multiple return values. Performance: Tuples are slightly faster (less overhead, fixed size). Hashability: Tuples can be dict keys/set members; lists cannot. In data pipelines: Tuples for schema-like rows (column names); lists for buffers, accumulators....

Question 5

Explain the difference between shallow copy and deep copy in Python.

Accepted Answer

Shallow (copy.copy()): New top-level object; nested objects are references. Nested mutations affect original. Deep (copy.deepcopy()): Recursive copy; fully independent. Why it matters: Shallow is O(n) for top level only; deep is O(n) for entire structure—can be slow for large nested dicts. Use shallow when: No nested mutables or shared refs OK. Use deep when: Need full isolation (e.g., config that will be modified)....

Question 6

Write a Python function to find the first non-repeating character in a string.

Accepted Answer

Approach: Two-pass—count chars, then find first with count 1. Code: def first_non_repeating(s): counts = {}; [counts.update({c: counts.get(c, 0) + 1}) for c in s]; return next((c for c in s if counts[c] == 1), None). Or: from collections import Counter; counts = Counter(s); return next((c for c in s if counts[c] == 1), None). Complexity: O(n) time, O(k) space. Why: Single pass can't know if char is unique until full scan....

Question 7

What are decorators in Python, and how do they work?

Accepted Answer

Decorator: Higher-order function that wraps another function; @decorator syntax. Implementation: def timer(f): def wrapper(*args, **kwargs): start=time(); r=f(*args,**kwargs); print(time()-start); return r; return wrapper. Why: Cross-cutting concerns—logging, retries, auth, caching—without cluttering business logic. In data pipelines: @retry, @logged, @rate_limited on API fetchers. Scalability: Decorators add indirection; overuse can make stack traces deep....

Question 8

Explain the difference between args and kwargs in Python.

Accepted Answer

*args: Variable positional args, received as tuple. **kwargs: Variable keyword args, received as dict. Why: Flexible signatures—wrappers, decorators, config-driven functions. Example: def run_pipeline(*sources, **config): ... allows run_pipeline('s3://a','s3://b', parallelism=4). Order: (regular, *args, keyword-only, **kwargs). Pitfall: Passing through—func(*args, **kwargs)—preserves interface. In data pipelines: **kwargs for optional connector params (region, timeout, retries)....

Question 9

What challenges did you face, and how did you tackle them?

Accepted Answer

Three key challenges: (1) Scaling: Pipeline throughput 10x'd while latency targets stayed fixed. Situation: Joins were the bottleneck. Action: I profiled with Spark UI, then applied broadcast joins for small dims, bucketing for large tables, and partitioned incremental processing. Result: 70% latency reduction. (2) Data Quality: Silent failures caused downstream confusion. Situation: No validation layer....

Question 10

What would you do if a pipeline failed and you couldn't find the reason?

Accepted Answer

Situation: A production pipeline fails; logs and metrics don't immediately reveal the cause. Task: Isolate the failure, restore service, and find root cause. Action: I follow a systematic approach. (1) Isolate: Identify failing stage, partition, and approximate record range from logs. (2) Reproduce: Run the same job on a subset in dev with verbose logging; use Spark local mode or a debugger if needed. (3) Bisect: Narrow the data range (e.g., single partition) to find the offending records....

Delivery Hero Data Engineer Interview Questions

Difficulty Breakdown

Key Topics Covered

How to Use This Guide

Companies asking these questions

All 31 Questions

More Interview Prep Guides

Unlock All Expert Answers