Question 1

What is the difference between cache() and persist() in Spark? When would you use each?

Accepted Answer

**cache()**: Equivalent to `persist(MEMORY_AND_DISK)`. Stores partitions in memory; spills to disk if memory is insufficient.

**persist(storage_level)**: Explicit control over storage: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY.

**Architectural Logic (Why It Matters)**: Caching trades memory/disk for recomputation cost....

Question 2

Demonstrate the difference between DENSE_RANK() and RANK()

Accepted Answer

**RANK()**: Same rank for ties; skips subsequent ranks (e.g., 1, 2, 2, 4, 5). **DENSE_RANK()**: Same rank for ties; no gaps (e.g., 1, 2, 2, 3, 4). **Why it matters**: RANK preserves "position" semantics (e.g., 4th place); DENSE_RANK gives consecutive integers useful for filtering (e.g., TOP 10). **Example**: `SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS rk, DENSE_RANK() OVER (ORDER BY salary DESC) AS dense_rk FROM employee`....

Question 3

Detail examples of inner, outer, left, and right joins.

Accepted Answer

**Architectural Logic**: Join choice directly impacts query cost, data correctness, and downstream semantics. INNER keeps only matching rows—ideal when referential integrity is enforced and you want to exclude orphans; minimizes shuffle and output size. LEFT preserves all from A with optional B—use when B is a "dimension" that may not exist (e.g., optional user attributes); RIGHT is mirror of LEFT; FULL OUTER preserves both sides—expensive due to bilateral NULL expansion....

Question 4

How would you handle a deadline conflict between two high-priority projects?

Accepted Answer

Situation: Two projects, same deadline. Task: Resolve with shared ownership. Action: Gathered facts; convened stakeholders and manager. Presented options (extend, split team, reduce scope). Chose scope reduction with sign-off....

Question 5

Discuss versioning in S3.

Accepted Answer

Architectural logic: Versioning stores multiple object versions per key; PUT creates new version; DELETE adds delete marker. Use cases: Recovery, retention, audit. Trade-off: Storage cost; lifecycle rules for old versions....

Question 6

Modify a word count script to output results in descending frequency order.

Accepted Answer

**Why Descending Order Matters:** Top-N by frequency is the core of term frequency analysis, log analysis, and recommendation features (most-viewed items).

**Scalability Tiers:** (1) Single file < 1GB: Counter + most_common()—in-memory, O(n log k) for top-k. (2) Multi-file/large: MapReduce pattern—map emits (word,1), reduce sums, final sort. (3) Spark: reduceByKey, then sortBy(col('count').desc())....

Question 7

What are docstrings? Use examples.

Accepted Answer

**Why Docstrings:** Documentation as code—accessible via help(), __doc__, and auto-generated APIs (Sphinx). Critical for team scale and onboarding.

**Formats:** Google, NumPy, reST. Include: purpose, Args, Returns, Raises, Examples. Type hints complement docstrings.

**Production:** We enforce docstrings on public functions. CI runs pydocstyle. Sphinx generates docs for our internal ETL lib....

Question 8

Which data structure occupies more memory: list or tuple? Why?

Accepted Answer

**Tuple < List:** Tuple is immutable—no over-allocation for growth. Simpler storage. sys.getsizeof((1,2,3)) < sys.getsizeof([1,2,3]). Typical 10–20% smaller for same elements.

**Why:** List has growth buffer; tuple fixed. CPython optimizes small tuples (interning).

**Production:** Use tuple for fixed records (return values, dict keys), list for mutable collections. In pipelines: tuple for (id, value) pairs; list for accumulating results....

Impetus Data Engineer Interview Questions

+12 More Questions with Expert Answers

Other Companies

Impetus Data Engineer Interview Questions

+12 More Questions with Expert Answers

Other Companies