Question 1

Design a fault-tolerant Spark Streaming checkpoint strategy: what to persist, recovery semantics, and cost/scalability trade-offs with checkpoint frequency.

Accepted Answer

**Section 1 — The Context (The 'Why')**
Spark Streaming fault tolerance requires checkpointing state and offsets. Checkpoint corruption loses replay; too-frequent checkpoints add overhead.

**Section 2 — The Diagram**
```
[Source] --> [Stream] --> [Sink]
  Checkpoint:S3  State:RocksDB
```

**Section 3 — Component Logic**
**Checkpoint** stores offsets and metadata to S3/HDFS. On restart, driver replays from last offset. **State store** (RocksDB) backs aggregation state....

Question 2

How do you handle version conflicts for libraries?

Accepted Answer

Situation: Our dbt project and a shared Spark job both depended on different PyArrow versions; one upgrade broke the other. We had 3 similar conflicts in a quarter. Task: Stabilize dependencies and prevent future conflicts. Action: I enforced lock files (poetry.lock, pip-tools) and hashed requirements.txt for reproducible builds. I created an internal package index with vetted, tested versions. I introduced a dependency review process—any upgrade required CI to pass for all consumers....

Question 3

How is Azure Key Vault used to manage encryption keys in Databricks?

Accepted Answer

Architectural flow: Key Vault stores secrets and keys. Databricks accesses via Azure AD (Managed Identity or SP). Use: DB passwords, API keys; dbutils.secrets.get(scope, key). Secret scope linked to Key Vault. CMK for workspace encryption....

Question 4

What are the differences between %run and dbutils.notebook.run?

Accepted Answer

**%run**: Executes notebook inline in the same Spark session. Variables and imports are shared. Child notebook can modify parent's state. Use for quick composition, shared setup (imports, config). **dbutils.notebook.run()**: Executes in separate context. No variable sharing. Returns last expression. Supports timeout and arguments. Use for job workflows, parallel runs, parameterized execution. **When to use which**: %run for development, ad-hoc composition....

Question 5

Can you describe the role of user groups in setting up these policies?

Accepted Answer

**Groups in RBAC**: Assign permissions to groups; add users to groups. Simplifies management—add user to analyst group vs. granting per-object. **Databricks**: Create group; grant on schema/table. Sync from IdP (Okta, Azure AD)....

Question 6

How do these transformations impact memory usage?

Accepted Answer

**Impact**: Shuffle (groupBy, join)—data redistribution; spill if OOM. collect()/toPandas()—pull to driver; avoid on large. cache()—uses memory/disk. Wide transformations—many output partitions. **Mitigation**: More partitions; broadcast small joins; avoid collect; streaming for large. Monitor Spark UI....

Question 7

How do you ensure version control when migrating notebooks?

Accepted Answer

**Approach**: (1) Export to .py or .ipynb; (2) Git—commit notebooks; use nbdime for diff; (3) Review—notebooks are JSON; (4) CI—papermill, nbval for tests; (5) No secrets—templates, inject at runtime. Pair notebooks with .py modules. Databricks Repos for sync....

Question 8

How do you handle passing parameters between notebooks?

Accepted Answer

**Databricks**: Widgets (dbutils.widgets.text); %run ./notebook $param=value. **Papermill**: papermill input.ipynb output.ipynb -p param value. **Shared state**: DBFS file or table. **Orchestrator**: Airflow passes params to tasks....

TCS Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies

TCS Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies