Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why Choose One Over the Other**: Latency SLA drives the decision. Analytics and ML training tolerate hours; fraud detection and alerts need seconds. **Batch (Spark)**: (1) Higher throughput per dollar—sequential reads, bulk writes. (2) Simpler failure model—rerun from last...
This hard-level Spark/Big Data question appears frequently in data engineering interviews at companies like PayPal. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (partition, spark) will help you answer variations of this question confidently.
This is a senior-level question that tests architectural thinking. Lead with the high-level design, then drill into specifics. Discuss trade-offs explicitly - there is rarely one correct answer. Show awareness of scale, fault tolerance, and operational complexity.
Why Choose One Over the Other: Latency SLA drives the decision. Analytics and ML training tolerate hours; fraud detection and alerts need seconds.
Batch (Spark): (1) Higher throughput per dollar—sequential reads, bulk writes. (2) Simpler failure model—rerun from last checkpoint. (3) Lower ops overhead—no 24/7 cluster. (4) Latency: minutes to hours.
Streaming (Kafka + Spark/Flink): (1) Sub-second to minute latency. (2) Event-driven; backpressure. (3) Exactly-once adds complexity. (4) Always-on clusters; higher cost.
Scalability Trade-offs: Batch scales with partition count and executor count. Streaming scales with Kafka partitions and consumer parallelism; mismatch causes lag.
Cost Implications: Per-GB processing: batch 5–10x cheaper. Streaming: 2–3x cluster cost for 24/7. Hybrid (Lambda): Streaming for speed layer, batch for batch layer; doubles storage and compute.
Pro-Move: Incremental batch (e.g., hourly micro-batches) as a compromise—lower latency than daily, simpler than full streaming.
Pro-Move: 'We use hourly micro-batch for reporting; streaming only for fraud—saved 40% infra.' Red Flag: Building streaming when batch SLA is sufficient—classic over-engineering.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.