Question 1

Describe how you would optimize slow-running Spark jobs in a distributed environment.

Accepted Answer

**Why systematic optimization matters**: Ad-hoc fixes don't scale; methodical profiling does. **Approach**: (1) Profile—Spark UI: stages, tasks, shuffle read/write. (2) Partition pruning and predicate pushdown. (3) Broadcast small tables. (4) Repartition for skew. (5) Cache reused DataFrames. (6) Tune shuffle partitions. (7) Enable AQE. (8) Right-size executors. **Scalability trade-offs**: Each lever has limits; combine for best effect....

Question 2

Explain your approach to monitoring and logging Spark jobs in AWS. What tools would you use to identify performance bottlenecks?

Accepted Answer

**Situation**: Faced competing demands—multiple pipelines, stakeholders, deadlines. **Task**: Deliver impact while maintaining quality and preventing burnout. **Action**: (1) Prioritized by business impact and SLA risk. (2) Used ROI (value/time); WIP limits; timeboxing. (3) Communicated trade-offs—'Adding X pushes Y by N days.' (4) Maintained backlog with tech-debt capacity. **Result**: Shipped on time; zero incidents; stakeholder alignment on deferrals....

Question 3

How do you implement incremental updates in a data lake using AWS services and Spark?

Accepted Answer

**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Incremental updates in data lake: (1) Track watermark (max timestamp/ID); (2) Read only new data from source; (3) MERGE into Delta or overwrite partitions; (4) Use Spark structured streaming or batch with checkpoint. Example: read S3 since last_run, merge to Delta....

EPAM Spark & Big Data Interview Questions

Companies asking these questions

All 3 Questions

More Interview Prep Guides

Get All Answers in PDF Format

EPAM Spark & Big Data Interview Questions

Companies asking these questions

All 3 Questions

More Interview Prep Guides

Get All Answers in PDF Format