Question 1

Design: Migrate data from multiple sources (Hadoop, S3, Oracle DB) into a final S3 bucket

Accepted Answer

**Section 1 — The Context (The 'Why')**
The primary challenge for this design in Cloud/Tools is balancing scale, cost, and reliability. At scale, naive approaches fail: single points of failure cause cascades, schema evolution breaks consumers, and over-provisioning explodes cost. Failure modes include silent data loss from non-idempotent writes, cascading job failures from tight coupling, and operational burden from manual intervention....

Question 2

Why specific cloud services (AWS Glue, EMR) were chosen for scalability and cost-effectiveness

Accepted Answer

**Glue**: Serverless Spark; no cluster management; Data Catalog integration; job bookmarks. Suits standard batch ETL, moderate Spark. Cost-effective for variable/infrequent jobs. 15-min DPU minimum—inefficient for very short jobs. **EMR**: Full Spark control; custom configs; MLlib, graph. Use when Glue doesn't fit—long-running, custom libs. **Scalability**: Both auto-scale; Glue per-job; EMR per-cluster. **Cost**: Glue for sporadic; EMR for steady or custom....

Question 3

Rainwater Trapping Problem - solve with two-pointer technique

Accepted Answer

Water at i = min(left_max, right_max) - height[i]. Two pointers: maintain left_max, right_max; move from side with smaller max. Python: l,r=0,len(h)-1; lm=rm=w=0; while l<r: if h[l]<h[r]: lm=max(lm,h[l]); w+=lm-h[l]; l+=1; else: rm=max(rm,h[r]); w+=rm-h[r]; r-=1; return w. O(n) time, O(1) space....

Question 4

Resolving service downtimes

Accepted Answer

(1) TRIAGE—Confirm scope, check dashboards, recent deploys. (2) MITIGATE—Rollback, scale up, disable non-critical features. (3) INVESTIGATE—Logs (CloudWatch, ELK), metrics, dependencies. (4) FIX—Patch, config, hotfix. (5) COMMUNICATE—Status page, stakeholders. (6) POST-MORTEM—Timeline, root cause, action items. WHY: Minimize MTTR; learn to prevent....

Question 5

Collaborating with cross-functional teams to resolve data quality issues

Accepted Answer

**Situation:** DQ issues affecting multiple teams. **Task:** Resolve with cross-functional alignment. **Action:** Define SLAs; automated checks (Deequ, Great Expectations); catalog + ownership; blameless postmortems; shared dashboards; embedded pipeline checks. **Result:** Trust via transparency; consistent delivery....

Question 6

Priority Queue Problem - task prioritization and dynamic sorting

Accepted Answer

**Why Priority Queues in Data Pipelines:** Task scheduling (Airflow, Celery), event processing (process by timestamp/priority), and merge of K sorted streams (heapq.merge).

**Implementations:** (1) heapq: min-heap, O(log n) push/pop. (2) queue.PriorityQueue: thread-safe wrapper. (3) For dynamic priority updates: need decrease_key—heapq doesn't support; use lazy deletion (mark stale, ignore when popped) or custom Fibonacci heap.

**Scalability:** Single-process heap = millions of items....

Question 7

Query Optimization Strategies

Accepted Answer

Query optimization: use indexes; avoid SELECT *; push filters early; use JOINs not subqueries; analyze execution plans; partition pruning; materialized views; statistics; limit result sets; avoid functions on indexed columns (e.g., WHERE YEAR(date)=2024 prevents index use—use range instead). **Why it matters**: Design choices compound at scale—wrong approach can cause 100× overhead. **Scalability trade-offs**: Profile before optimizing; validate on sample then full....

Question 8

Given two DataFrames, perform specified data transformations and store the result in a new DataFrame

Accepted Answer

**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Given df1, df2: common transformations—join (df1.join(df2, 'key')), union (df1.union(df2)), concat (unionByName with allowMissingColumns). Then apply filters, aggregates, select. Example: result = df1.join(df2, 'id').select(df1.id, df1.name, df2.score).filter(col('score')>70)....

Question 9

Trade-offs between batch processing (Spark) vs. real-time streams (Kafka)

Accepted Answer

**Why Choose One Over the Other**: Latency SLA drives the decision. Analytics and ML training tolerate hours; fraud detection and alerts need seconds.

**Batch (Spark)**: (1) Higher throughput per dollar—sequential reads, bulk writes. (2) Simpler failure model—rerun from last checkpoint. (3) Lower ops overhead—no 24/7 cluster. (4) Latency: minutes to hours.

**Streaming (Kafka + Spark/Flink)**: (1) Sub-second to minute latency. (2) Event-driven; backpressure. (3) Exactly-once adds complexity....

Question 10

Handling pipeline overload situations

Accepted Answer

**Section 1 — The Context (The 'Why')**
This system design addresses "Handling pipeline overload situations..." at production scale. The primary challenges are throughput under variable load, fault tolerance across distributed components, and maintaining consistency guarantees that match business requirements. Naive monolithic approaches break when single points of failure emerge, network partitions occur, or backpressure from slow consumers cascades upstream....

PayPal Data Engineer Interview Questions

Difficulty Breakdown

Key Topics Covered

How to Use This Guide

Companies asking these questions

All 10 Questions

More Interview Prep Guides

Unlock All Expert Answers