Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Watermark: Defines max lateness (e.g., 10 min). Events older than (max_event_time - watermark) are dropped. State: Kept for aggregations within watermark; beyond that, state is purged to avoid unbounded growth. Output modes: Append (only finalized results), Update (changed...
Red Flag: Saying 'use watermark' without discussing state growth or dropped data handling. Pro-Move: 'We use 10-min watermark aligned to p99 latency; late events go to DLQ for batch backfill'—shows end-to-end design.
This hard-level Spark/Big Data question appears frequently in data engineering interviews at companies like Bitwise, Incedo, Swiggy, and 1 others. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (spark, window) will help you answer variations of this question confidently.
This is a senior-level question that tests architectural thinking. Lead with the high-level design, then drill into specifics. Discuss trade-offs explicitly - there is rarely one correct answer. Show awareness of scale, fault tolerance, and operational complexity.
Watermark: Defines max lateness (e.g., 10 min). Events older than (max_event_time - watermark) are dropped. State: Kept for aggregations within watermark; beyond that, state is purged to avoid unbounded growth. Output modes: Append (only finalized results), Update (changed rows), Complete (full result). Why: Trade-off between latency and correctness—tighter watermark = less state, more dropped late data. Looser = more state, more complete results. Scalability: State scales with unique keys × window count; oversized watermark = OOM risk. Cost: Longer watermark = more state = more memory. Best practice: Set watermark from SLA (e.g., 99% of events arrive within 5 min). Use DLQ for dropped events when audit is needed.
Red Flag: Saying 'use watermark' without discussing state growth or dropped data handling. Pro-Move: 'We use 10-min watermark aligned to p99 latency; late events go to DLQ for batch backfill'—shows end-to-end design.
Practice the 66 most asked data engineering questions at Swiggy. Covers SQL, Spark/Big Data, Python/Coding and more.
13 min read →Practice the 44 most asked data engineering questions at Incedo. Covers Spark/Big Data, SQL, Behavioral and more.
8 min read →Senior Spark interviews at Amazon, Databricks, and Meta focus on performance tuning, not API syntax. Master these 15 questions to prove you've run Spark at scale.
20 min read →Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 4 companies. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.