Real Apache Kafka and streaming questions from data engineering interviews — topics, partitions, consumer groups, and delivery semantics.
Apache Kafka is the default backbone for real-time data pipelines, and streaming rounds increasingly appear in data engineering loops. These questions cover Kafka's log architecture, topics and partitioning, consumer groups and rebalancing, delivery guarantees (at-least-once vs exactly-once), retention and compaction, and how Kafka fits into streaming ETL. Each question includes a detailed answer.
This collection contains 50 curated questions: 5 easy, 2 medium, and 43 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are spark (30), partition (19), optimization (12), window (10), etl (9), and join (7). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
Tell me about yourself and your experience.
What is the difference between SparkSession and SparkContext in Spark?
CDC During Migration - explain approaches for real-time Change Data Capture
Explain the types of triggers in ADF, including schedule, tumbling window, and event-based triggers.
How do you handle late-arriving data in Spark Structured Streaming?
Briefly explain the architecture of Kafka.
Describe the data pipeline architecture you've worked with.
Design a fault-tolerant Spark Streaming checkpoint strategy: what to persist, recovery semantics, and cost/scalability trade-offs with checkpoint frequency.
Difference Between Internal and External Tables in BigQuery
Explain the difference between batch and streaming data processing in Data Fusion.
Explain the trade-offs between batch and real-time data processing. Provide examples of when each is appropriate.
Given a streaming dataset from Kafka, how would you ingest the data in real-time using Spark?
How would you implement a sliding window aggregation in Spark Structured Streaming?
What are the key components of AWS Glue, and how do they work together?
What is the difference between a generator and a list in Python?
What is the small-file problem in Spark, and how do you solve it?
Why are you leaving your current company?
Why do you want to join this company?
Why is SparkSession used in Spark 2.0 and later versions?
Apache Spark Fundamentals - discuss
Briefly introduce yourself and walk us through your journey as a Data Engineer so far.
Can Presto work with Near Real-Time Data (Streaming Data Source)?
Can you elaborate on your Big Data project experience?
Can you explain how streams and tasks handle data freshness in near real-time?
Can you give a use case where Delta Live Tables would be ideal?
Command to Read JSON Data and Options
Compare batch processing and stream processing for financial data.
Compare Hadoop and Spark. Which one would you choose for a real-time application, and why?
Compare Kafka and RabbitMQ for real-time message processing in a streaming platform.
Compare Kafka Streams and Spark Structured Streaming for real-time processing
Conceptualize and design a real-time streaming data pipeline end-to-end.
Core services of AWS used in data engineering?
Count occurrences of a specific word in a file
Databricks - platform, use cases
Delta Lake: ACID compliance, time travel, streaming support
Describe a project you worked on, focusing on the data pipeline and your role.
Describe a recent project where you used AWS services extensively. What was your role, and what challenges did you face?
Describe a scenario where you used Databricks for real-time data processing.
Describe a strategy for implementing a real-time content delivery monitoring system.
Describe a time when you had to work with a team to solve a complex problem.
Describe a time you had to learn a new technology quickly to solve a problem.
Describe a time you had to make a difficult decision with limited information.
Describe a time you had to make a difficult decision with limited information.
Describe a time you had to work with a difficult stakeholder.
Describe AWS Glue components and their functions.
Describe how Kafka ensures data durability and fault tolerance.
Describe how to monitor and log errors effectively in a real-time data pipeline.
Describe how you would architect a pipeline to process real-time logs with schema evolution
Describe how you'd design a system to track inventory and sales in real-time.
Describe the projects emphasizing Spark, Hadoop, or Azure for large-scale data processing
Kafka is a distributed, durable, high-throughput event streaming platform. It decouples producers from consumers through an append-only, replayable log, letting many independent systems read the same stream at their own pace. Data engineers use it to ingest high-volume event data (clicks, logs, IoT, CDC), buffer spikes, and feed both real-time and batch pipelines — the backbone of most modern streaming architectures.
A topic is a named stream of records. Each topic is split into partitions, which are the unit of parallelism and ordering — records within a partition are strictly ordered and identified by a monotonically increasing offset. Producers write to partitions (by key hash or round-robin); consumers track their offset per partition to know what they've read. More partitions means more parallelism but also more overhead.
A consumer group is a set of consumers that cooperatively read a topic — each partition is assigned to exactly one consumer in the group, so work is divided and scaled horizontally. When a consumer joins or leaves (or a partition is added), Kafka triggers a rebalance that reassigns partitions across the remaining consumers. Rebalancing briefly pauses consumption, so minimizing it (via static membership, cooperative rebalancing) matters at scale.
By default Kafka gives at-least-once delivery — a consumer may reprocess a record if it fails before committing its offset, so consumers should be idempotent. At-most-once is possible by committing offsets before processing (risking loss). Exactly-once semantics (EOS) is achieved with idempotent producers plus transactions that atomically write records and commit offsets, ensuring each record affects the output exactly once — essential for financial or dedup-sensitive pipelines.
Traditional queues typically delete a message once it's consumed and push to one consumer. Kafka is a durable, replayable log: messages are retained for a configurable period and multiple consumer groups can read the same data independently, at their own offset. This makes Kafka ideal for event streaming, replay, and fan-out to many downstream systems, whereas classic queues suit simple task distribution and work queues.
Hands-on SQL, Python, and pipeline courses — build the skills interviewers test.
A structured, resume-boosting certificate covering the full DE stack.
The book that gets data engineers through system-design rounds. Essential reading.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
Reading answers is step one. Get instant AI feedback on your answers, run mock interviews, and track readiness — built specifically for data engineering interviews.