Kafka & Real-Time Streaming Interview Questions (2026)
Apache Kafka and real-time streaming architectures power modern event-driven data platforms. Interviewers evaluate your mastery of partition assignment strategies, consumer group rebalancing, leader election, log compaction, offset commit semantics (at-least-once vs exactly-once), and stream processing engines like Apache Flink.
Tested Core Concepts & Weightage
Distribution within this topicUnlocked FAANG-Calibrated Answer Rubric (Top 3 Tested Questions)
- •Rigorous explanation of data distribution, memory boundaries, and execution bottlenecks
- •Quantified improvements (e.g. 60% shuffle volume reduction, predictable SLA under 10x spikes)
- •Proactive trade-off analysis between latency, storage overhead, and operational complexity
Complete Kafka & Real-Time Streaming Question Vault (80 Questions)
Showing 15 of 80Design a fault-tolerant Spark Streaming checkpoint strategy: what to persist, recovery semantics, and cost/scalability trade-offs with checkpoint frequency.
**Section 1 — The Context (The 'Why')** Spark Streaming fault tolerance requires checkpointing state and offsets. Checkpoint corruption loses replay; too-frequent checkpoints add overhead. **Section 2 — The Diagram** ``` [Source] --> [Stream] --> [Sink] Checkpoint:S3 State:RocksDB ``` **Section 3 — Component Logic** **Checkpoint** stores offsets and metadata to S3/HDFS. On restart, driver replays from last offset. **State store** (RocksDB) backs aggregation state....
Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.
**Pattern**: Process only new/changed data by tracking last processed boundary (watermark) or using CDC. **ADF approach**: Watermark via lookup/stored procedure storing max(modified_date); filter source WHERE modified_date > @lastRun; parameterize pipeline for lastRun; write to sink. Use triggers for scheduling. **Databricks approach**: Delta Lake MERGE INTO for upserts; Change Data Feed (CDF) for CDC; read CDF or filter on _change_version....
Explain the difference between batch and streaming data processing in Data Fusion.
Batch processes bounded datasets in discrete runs; streaming processes unbounded data with continuous execution. **Why the distinction matters**: Batch has predictable cost (run N times, pay N × job cost); streaming has always-on cost and state management. **Data Fusion context**: Batch templates (e.g., JDBC to BigQuery) are scheduled; streaming uses Pub/Sub or Kafka sources....
Given a streaming dataset from Kafka, how would you ingest the data in real-time using Spark?
Kafka ingestion with Spark Structured Streaming follows a standard pattern: readStream → parse → writeStream with checkpoint. **Architectural decisions**: (1) **startingOffsets**: 'earliest' for backfill, 'latest' for tail-only; use JSON per-partition offsets for exactly-once replay. (2) **Checkpoint**: Mandatory for exactly-once; stores offsets + write metadata; without it, duplicates or data loss on restart....
How would you implement a sliding window aggregation in Spark Structured Streaming?
Sliding window: window(timeColumn, windowDuration, slideDuration) where slideDuration < windowDuration creates overlapping windows. Example: df.withWatermark("event_time", "10 minutes").groupBy(window(col("event_time"), "1 hour", "10 minutes"), col("user_id")).count(). **Why watermark**: Late data would grow state unbounded; watermark drops events older than (max_event_time - delay) and allows state cleanup....
Describe how Adidas could use S3 and Athena to analyze clickstream data.
Architecture: Ingestion via API Gateway + Lambda or Kinesis → S3 landing zone (JSON/Parquet) partitioned dt=YYYY-MM-DD. Glue Crawlers or manual schema → Athena tables. Query: Funnels, sessions, A/B tests—e.g., conversion by landing page. Why S3 + Athena: Decoupled storage/compute; pay per query; no cluster. Scalability: S3 unlimited; Athena concurrency unlimited. Cost: Partition by date and campaign_id; use Parquet—10x compression, column pruning....
How do you ensure message ordering in Kinesis Streams?
Architectural logic: Ordering is per partition key. Same key → same shard → ordered. Use consistent partition key (user_id, order_id) for causal ordering. Trade-off: Same key = potential hot partition. Best practice: Partition by entity ID; multiple keys for parallelism.
How does ADF help streamline data movement in your project?
Benefits: (1) 100+ connectors. (2) SHIR for on-prem. (3) Copy and Data Flows. (4) Scheduling, triggers. (5) Parameters for reuse. Example: One pipeline for SAP, Salesforce, on-prem SQL → lake; another to Synapse....
What alternatives to Kinesis would you consider for real-time data ingestion?
**Decision framework**: Throughput, latency, replay, operational model, ecosystem. **Kafka (MSK)**: Higher throughput, partition replay, Kafka ecosystem (Connect, Streams). Use when you need exactly-once semantics, consumer lag monitoring, or multi-sink patterns. **Apache Pulsar**: Multi-tenancy, tiered storage, geo-replication. Use for multi-tenant SaaS. **Google Pub/Sub**: Serverless, at-least-once. Simpler ops; no partition management....
What metrics would you track in CloudWatch for a Kinesis-based pipeline?
**Kinesis metrics**: GetRecords.IteratorAgeMilliseconds (lag—critical; target <1 min for real-time); IncomingBytes, IncomingRecords; PutRecord success/failure. **Lambda (consumer)**: Duration, Errors, Throttles, ConcurrentExecutions. **End-to-end**: IteratorAge indicates consumer lag—if growing, consumers can't keep up. **Alarms**: IteratorAge > 60 sec; Errors > 0; Throttles > 0. **Dashboard**: Lag, throughput (bytes/records per sec), error rate, Lambda duration....
Compare batch processing and stream processing for financial data.
Batch processing handles large volumes of historical financial data at scheduled intervals for tasks like reporting and reconciliation, prioritizing throughput. Stream processing, conversely,…
Designing Mixpanel - event-driven analytics platform
**Section 1 — The Context (The 'Why')** The primary challenge for this design in General/Other is balancing scale, cost, and reliability. At scale, naive approaches fail: single points of failure cause cascades, schema evolution breaks consumers, and over-provisioning explodes cost. Failure modes include silent data loss from non-idempotent writes, cascading job failures from tight coupling, and operational burden from manual intervention....
Explain how to implement schema validation for incoming data streams.
**Why**: Invalid records corrupt downstream; validation at ingress isolates failures. **Components**: (1) Schema Registry (Avro/Proto/JSON Schema)—versioned schemas; (2) Validate at ingress—Kafka with Schema Registry, API gateway; (3) Check required fields, types, enums; (4) Dead-letter queue for invalid. **Evolution**: Backward/forward compatible changes. Confluent Schema Registry; producers validate before produce. For JSON: jsonschema, pydantic....
Handle midstream schema changes gracefully.
**Strategy**: (1) Schema evolution—additive changes; new columns, defaults for old; (2) Schema Registry—versioned, backward/forward compatible; (3) Multiple readers—support versions during transition; (4) Dead-letter for incompatible. **Parquet/Delta**: mergeSchema=True. Avro: Schema Registry. Design for evolution; document breaking vs additive; compatibility tests in CI....
Design an algorithm to merge k sorted lists of video streaming data.
**Section 1 — The Context (The 'Why')** The primary challenge for this design in Python/Coding is balancing scale, cost, and reliability. At scale, naive approaches fail: single points of failure cause cascades, schema evolution breaks consumers, and over-provisioning explodes cost. Failure modes include silent data loss from non-idempotent writes, cascading job failures from tight coupling, and operational burden from manual intervention....
Stop Guessing If Your Answers Clear the Bar at Kafka & Real-Time Streaming
DataEngPrep’s AI evaluates your real answers against strict FAANG hire rubrics. Check accuracy, optimize shuffle partitions, detect edge case omissions, and receive an instant 9/10 rewrite before stepping into your real interview loop.
Related Technical Topics & Competencies
Frequently paired together during multi-round senior data engineering loops.
Frequently Asked Questions About Kafka & Real-Time Streaming Interviews
How does Kafka guarantee message ordering?▼
Kafka guarantees total ordering of messages *only within a single partition*. Across multiple partitions, messages are processed concurrently without global ordering. If strict ordering is required, producer messages must share the same partition key.
What causes consumer group rebalancing and how do you prevent rebalance storms?▼
Rebalances occur when a consumer joins/leaves, crashes, or fails to send heartbeats within `session.timeout.ms`, or when processing a poll batch takes longer than `max.poll.interval.ms`. To prevent storms, tune `max.poll.records` down, use cooperative sticky assignors, and avoid blocking operations inside the consumer loop.
How does Kafka achieve Exactly-Once Semantics (EOS)?▼
Kafka EOS combines idempotent producers (using sequence numbers and producer IDs to eliminate duplicates at the broker level) with transaction coordinators that atomically commit writes across multiple partitions and consumer offsets.
What real-time streaming questions appear most frequently in senior loops?▼
Popular questions include: handling out-of-order events using watermarks in Flink/Spark, designing change data capture (CDC) with Debezium and Kafka, and managing schema evolution with Schema Registry.