Question 1

What is the difference between cache() and persist() in Spark? When would you use each?

Accepted Answer

**cache()**: Equivalent to `persist(MEMORY_AND_DISK)`. Stores partitions in memory; spills to disk if memory is insufficient.

**persist(storage_level)**: Explicit control over storage: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY.

**Architectural Logic (Why It Matters)**: Caching trades memory/disk for recomputation cost....

Question 2

Demonstrate the difference between DENSE_RANK() and RANK()

Accepted Answer

**RANK()**: Same rank for ties; skips subsequent ranks (e.g., 1, 2, 2, 4, 5). **DENSE_RANK()**: Same rank for ties; no gaps (e.g., 1, 2, 2, 3, 4). **Why it matters**: RANK preserves "position" semantics (e.g., 4th place); DENSE_RANK gives consecutive integers useful for filtering (e.g., TOP 10). **Example**: `SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS rk, DENSE_RANK() OVER (ORDER BY salary DESC) AS dense_rk FROM employee`....

Question 3

Detail examples of inner, outer, left, and right joins.

Accepted Answer

**Architectural Logic**: Join choice directly impacts query cost, data correctness, and downstream semantics. INNER keeps only matching rows—ideal when referential integrity is enforced and you want to exclude orphans; minimizes shuffle and output size. LEFT preserves all from A with optional B—use when B is a "dimension" that may not exist (e.g., optional user attributes); RIGHT is mirror of LEFT; FULL OUTER preserves both sides—expensive due to bilateral NULL expansion....

Question 4

How would you handle a deadline conflict between two high-priority projects?

Accepted Answer

Situation: Two projects, same deadline. Task: Resolve with shared ownership. Action: Gathered facts; convened stakeholders and manager. Presented options (extend, split team, reduce scope). Chose scope reduction with sign-off....

Question 5

Discuss versioning in S3.

Accepted Answer

Architectural logic: Versioning stores multiple object versions per key; PUT creates new version; DELETE adds delete marker. Use cases: Recovery, retention, audit. Trade-off: Storage cost; lifecycle rules for old versions....

Question 6

Modify a word count script to output results in descending frequency order.

Accepted Answer

**Why Descending Order Matters:** Top-N by frequency is the core of term frequency analysis, log analysis, and recommendation features (most-viewed items).

**Scalability Tiers:** (1) Single file < 1GB: Counter + most_common()—in-memory, O(n log k) for top-k. (2) Multi-file/large: MapReduce pattern—map emits (word,1), reduce sums, final sort. (3) Spark: reduceByKey, then sortBy(col('count').desc())....

Question 7

What are docstrings? Use examples.

Accepted Answer

**Why Docstrings:** Documentation as code—accessible via help(), __doc__, and auto-generated APIs (Sphinx). Critical for team scale and onboarding.

**Formats:** Google, NumPy, reST. Include: purpose, Args, Returns, Raises, Examples. Type hints complement docstrings.

**Production:** We enforce docstrings on public functions. CI runs pydocstyle. Sphinx generates docs for our internal ETL lib....

Question 8

Which data structure occupies more memory: list or tuple? Why?

Accepted Answer

**Tuple < List:** Tuple is immutable—no over-allocation for growth. Simpler storage. sys.getsizeof((1,2,3)) < sys.getsizeof([1,2,3]). Typical 10–20% smaller for same elements.

**Why:** List has growth buffer; tuple fixed. CPython optimizes small tuples (interning).

**Production:** Use tuple for fixed records (return values, dict keys), list for mutable collections. In pipelines: tuple for (id, value) pairs; list for accumulating results....

Impetus Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies

Impetus Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies