Question 1

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Accepted Answer

**Narrow transformations**: Each input partition maps to at most one output partition. No shuffle. Examples: map, filter, flatMap, mapPartitions.

**Wide transformations**: Require data from multiple input partitions to produce one output partition. Trigger shuffle. Examples: groupByKey, reduceByKey, join, distinct, repartition.

**Architectural Logic (Why This Matters)**: Spark pipelines narrow transformations and executes them in a single stage....

Question 2

Explain the difference between Spark's map() and flatMap() transformations.

Accepted Answer

map(): 1-to-1; each input element yields one output. flatMap(): 1-to-many or 1-to-0; function returns iterable, results flattened. Example: map splits string → one string; flatMap splits → multiple words. Why it matters: map preserves partition count; flatMap can change it (e.g., explode JSON array → more rows). Use flatMap for: tokenization, exploding arrays, filtering (return empty list). Performance: flatMap that expands heavily can cause skew; monitor partition sizes....

Question 3

Explain the concept of Broadcast Join in Spark. When should it be used?

Accepted Answer

Mechanism: Small table sent to all executors; join happens locally, no shuffle. Triggered by broadcast() hint or spark.sql.autoBroadcastJoinThreshold (default 10MB). Why: Shuffle of large table is expensive; broadcasting small table avoids it. When: One side fits in executor memory (~broadcast threshold). Trade-off: Too large = driver/executor OOM; too small threshold = unnecessary shuffles. Cost: Broadcast data replicated per executor; acceptable for MB-scale....

Question 4

What is the difference between a list and a tuple in Python?

Accepted Answer

List: Mutable, []; tuple: immutable, (). Why it matters: Mutability drives use—lists for collections that change; tuples for fixed data, dict keys (hashable), multiple return values. Performance: Tuples are slightly faster (less overhead, fixed size). Hashability: Tuples can be dict keys/set members; lists cannot. In data pipelines: Tuples for schema-like rows (column names); lists for buffers, accumulators....

Question 5

Explain the difference between shallow copy and deep copy in Python.

Accepted Answer

Shallow (copy.copy()): New top-level object; nested objects are references. Nested mutations affect original. Deep (copy.deepcopy()): Recursive copy; fully independent. Why it matters: Shallow is O(n) for top level only; deep is O(n) for entire structure—can be slow for large nested dicts. Use shallow when: No nested mutables or shared refs OK. Use deep when: Need full isolation (e.g., config that will be modified)....

Question 6

Write a Python function to find the first non-repeating character in a string.

Accepted Answer

Approach: Two-pass—count chars, then find first with count 1. Code: def first_non_repeating(s): counts = {}; [counts.update({c: counts.get(c, 0) + 1}) for c in s]; return next((c for c in s if counts[c] == 1), None). Or: from collections import Counter; counts = Counter(s); return next((c for c in s if counts[c] == 1), None). Complexity: O(n) time, O(k) space. Why: Single pass can't know if char is unique until full scan....

Question 7

What are decorators in Python, and how do they work?

Accepted Answer

Decorator: Higher-order function that wraps another function; @decorator syntax. Implementation: def timer(f): def wrapper(*args, **kwargs): start=time(); r=f(*args,**kwargs); print(time()-start); return r; return wrapper. Why: Cross-cutting concerns—logging, retries, auth, caching—without cluttering business logic. In data pipelines: @retry, @logged, @rate_limited on API fetchers. Scalability: Decorators add indirection; overuse can make stack traces deep....

Question 8

Explain the difference between args and kwargs in Python.

Accepted Answer

*args: Variable positional args, received as tuple. **kwargs: Variable keyword args, received as dict. Why: Flexible signatures—wrappers, decorators, config-driven functions. Example: def run_pipeline(*sources, **config): ... allows run_pipeline('s3://a','s3://b', parallelism=4). Order: (regular, *args, keyword-only, **kwargs). Pitfall: Passing through—func(*args, **kwargs)—preserves interface. In data pipelines: **kwargs for optional connector params (region, timeout, retries)....

Delivery Hero Data Engineer Interview Questions

+2 More Questions with Expert Answers

Other Companies

Delivery Hero Data Engineer Interview Questions

+2 More Questions with Expert Answers

Other Companies