Question 1

Design a fault-tolerant Spark Streaming checkpoint strategy: what to persist, recovery semantics, and cost/scalability trade-offs with checkpoint frequency.

Accepted Answer

**Section 1 — The Context (The 'Why')**
Spark Streaming fault tolerance requires checkpointing state and offsets. Checkpoint corruption loses replay; too-frequent checkpoints add overhead.

**Section 2 — The Diagram**
```
[Source] --> [Stream] --> [Sink]
  Checkpoint:S3  State:RocksDB
```

**Section 3 — Component Logic**
**Checkpoint** stores offsets and metadata to S3/HDFS. On restart, driver replays from last offset. **State store** (RocksDB) backs aggregation state....

Question 2

Given a streaming dataset from Kafka, how would you ingest the data in real-time using Spark?

Accepted Answer

Kafka ingestion with Spark Structured Streaming follows a standard pattern: readStream → parse → writeStream with checkpoint. **Architectural decisions**: (1) **startingOffsets**: 'earliest' for backfill, 'latest' for tail-only; use JSON per-partition offsets for exactly-once replay. (2) **Checkpoint**: Mandatory for exactly-once; stores offsets + write metadata; without it, duplicates or data loss on restart....

Question 3

After cleaning, how would you store the transformed data into Delta Lake?

Accepted Answer

df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").save("/path/to/delta"). For upserts: DeltaTable merge. For append: mode("append"). Use partitionBy for query optimization; Z-ordering for filtered columns. **Why**: ACID, time travel, schema evolution. **Production**: delta.autoOptimize.optimizeWrite; run OPTIMIZE and VACUUM periodically. **Scalability**: Partition and Z-order for prune....

Question 4

Compare Kafka Streams and Spark Structured Streaming for real-time processing

Accepted Answer

**Why comparison matters**: Different operational models and best-fit use cases. **Kafka Streams**: Lightweight library; JVM; exactly-once; stateful; embeds in app. Good for microservices, low operational footprint. **Spark Structured Streaming**: Unified batch/stream API; rich ecosystem; larger footprint; micro-batch or continuous. Good for complex ETL, ML integration, team familiar with Spark. **Scalability trade-offs**: Kafka Streams = scale with app instances; Spark = scale with cluster....

Question 5

Databricks Cluster Management - standalone vs YARN mode

Accepted Answer

**Why mode matters**: Resource allocation and multi-tenancy differ. **Standalone**: Spark's built-in manager; simple; single-tenant. **YARN**: Hadoop resource manager; multi-tenant; queue management; enterprise. **Databricks**: Managed Spark; no direct YARN; uses cloud compute. On-prem Databricks can use YARN. **Scalability trade-offs**: Standalone = dev/simple; YARN = shared clusters, fairness. **Cost implications**: YARN enables resource sharing; standalone = dedicated....

Question 6

Design an ETL pipeline using Kafka and Spark Streaming

Accepted Answer

**Section 1 — The Context (The 'Why')**
ETL pipelines combining Kafka and Spark Streaming must reconcile batch-oriented processing with continuous ingestion. The primary challenge: offset management—without checkpoints, a Spark job restart replays from the beginning or skips data. Another failure mode is duplicate writes from at-least-once delivery when the sink is not idempotent....

Question 7

Explain how spark.read.format("delta").load() works

Accepted Answer

**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

spark.read.format('delta').load(path) reads a Delta table. Format delta uses the Delta Lake connector to (1) read the transaction log (_delta_log) to get current version; (2) determine which Parquet files are current; (3) read only those files (no full scan). Supports time travel: .option('versionAsOf', 5) or .option('timestampAsOf', 'timestamp')....

Question 8

Explain the architecture and role of the Hive Metastore in a data pipeline

Accepted Answer

**Section 1 — The Context (The 'Why')**
The Hive Metastore is the central schema registry for Hive, Spark, Presto, and other engines. A single point of failure, it blocks all schema operations when down—create table, alter, partition discovery. Running without a backup means schema loss on DB corruption....

Question 9

Explain the architecture of Kafka

Accepted Answer

**Section 1 — The Context (The 'Why')**
Kafka's architecture centers on partitioned, replicated logs—each partition has a leader and in-sync replicas (ISR). The primary failure modes are under-replicated partitions (data loss risk) and consumer lag (backpressure)....

Question 10

Explain the architecture of Spark Streaming

Accepted Answer

**Section 1 — The Context (The 'Why')**
Spark Streaming uses micro-batches. Exactly-once requires checkpoint + idempotent sink. DStreams deprecated. A naive pipeline checkpoints to local disk and uses non-idempotent sink.

**Section 3 — Component Logic**
**Receiver** fetches data. **Micro-batch** at trigger intervals....

Meesho Spark & Big Data Interview Questions

Companies asking these questions

All 26 Questions

More Interview Prep Guides

Get All Answers in PDF Format

Meesho Spark & Big Data Interview Questions

Companies asking these questions

All 26 Questions

More Interview Prep Guides

Get All Answers in PDF Format