Real Apache Kafka and streaming questions from data engineering interviews — topics, partitions, consumer groups, and delivery semantics.
Apache Kafka is the default backbone for real-time data pipelines, and streaming rounds increasingly appear in data engineering loops. These questions cover Kafka's log architecture, topics and partitioning, consumer groups and rebalancing, delivery guarantees (at-least-once vs exactly-once), retention and compaction, and how Kafka fits into streaming ETL. Each question includes a detailed answer.
This collection contains 34 curated questions: 2 easy, 2 medium, and 30 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are spark (20), partition (17), optimization (13), window (11), join (8), and python (3). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
Tell me about yourself and your experience.
What is the difference between SparkSession and SparkContext in Spark?
CDC During Migration - explain approaches for real-time Change Data Capture
Explain the types of triggers in ADF, including schedule, tumbling window, and event-based triggers.
How do you handle late-arriving data in Spark Structured Streaming?
Briefly explain the architecture of Kafka.
Describe the data pipeline architecture you've worked with.
Design a fault-tolerant Spark Streaming checkpoint strategy: what to persist, recovery semantics, and cost/scalability trade-offs with checkpoint frequency.
Difference Between Internal and External Tables in BigQuery
Explain the difference between batch and streaming data processing in Data Fusion.
Explain the trade-offs between batch and real-time data processing. Provide examples of when each is appropriate.
Given a streaming dataset from Kafka, how would you ingest the data in real-time using Spark?
How would you implement a sliding window aggregation in Spark Structured Streaming?
What are the key components of AWS Glue, and how do they work together?
What is the difference between a generator and a list in Python?
What is the small-file problem in Spark, and how do you solve it?
Why are you leaving your current company?
Why do you want to join this company?
Why is SparkSession used in Spark 2.0 and later versions?
Briefly introduce yourself and walk us through your journey as a Data Engineer so far.
Can Presto work with Near Real-Time Data (Streaming Data Source)?
Conceptualize and design a real-time streaming data pipeline end-to-end.
Describe a strategy for implementing a real-time content delivery monitoring system.
Describe a time when you had to work with a team to solve a complex problem.
Describe a time you had to learn a new technology quickly to solve a problem.
Describe a time you had to make a difficult decision with limited information.
Describe a time you had to make a difficult decision with limited information.
Describe a time you had to work with a difficult stakeholder.
Describe how Kafka ensures data durability and fault tolerance.
Describe how to monitor and log errors effectively in a real-time data pipeline.
Describe how you would architect a pipeline to process real-time logs with schema evolution
Describe how you'd design a system to track inventory and sales in real-time.
Design a data pipeline for real-time analytics of e-commerce transactions. Ensure to include data ingestion, processing, storage, and visualization components.
Design a data pipeline for streaming analytics.
Kafka is a distributed, durable, high-throughput event streaming platform. It decouples producers from consumers through an append-only, replayable log, letting many independent systems read the same stream at their own pace. Data engineers use it to ingest high-volume event data (clicks, logs, IoT, CDC), buffer spikes, and feed both real-time and batch pipelines — the backbone of most modern streaming architectures.
A topic is a named stream of records. Each topic is split into partitions, which are the unit of parallelism and ordering — records within a partition are strictly ordered and identified by a monotonically increasing offset. Producers write to partitions (by key hash or round-robin); consumers track their offset per partition to know what they've read. More partitions means more parallelism but also more overhead.
A consumer group is a set of consumers that cooperatively read a topic — each partition is assigned to exactly one consumer in the group, so work is divided and scaled horizontally. When a consumer joins or leaves (or a partition is added), Kafka triggers a rebalance that reassigns partitions across the remaining consumers. Rebalancing briefly pauses consumption, so minimizing it (via static membership, cooperative rebalancing) matters at scale.
By default Kafka gives at-least-once delivery — a consumer may reprocess a record if it fails before committing its offset, so consumers should be idempotent. At-most-once is possible by committing offsets before processing (risking loss). Exactly-once semantics (EOS) is achieved with idempotent producers plus transactions that atomically write records and commit offsets, ensuring each record affects the output exactly once — essential for financial or dedup-sensitive pipelines.
Traditional queues typically delete a message once it's consumed and push to one consumer. Kafka is a durable, replayable log: messages are retained for a configurable period and multiple consumer groups can read the same data independently, at their own offset. This makes Kafka ideal for event streaming, replay, and fan-out to many downstream systems, whereas classic queues suit simple task distribution and work queues.
The Data Engineering Interview Answer Vault bundles 750+ reviewed answers into 7 focused PDF volumes — SQL, Spark, Python, System Design, Cloud, Behavioral, and Data Modeling. Study on any device, no subscription required.
800+ hands-on courses — Grokking System Design, Coding Patterns, and AI mock interviews for your DE loop.
Turn any topic or your own notes into an interactive, personalized course in 60 seconds.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.