DataEngPrep.tech

JavaScript is required to use this application. Please enable JavaScript in your browser settings or disable any extensions that may be blocking scripts.

DataEngPrep.tech

Questions Blog Store

Home/Questions/Spark/Big Data/Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.

Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.

Spark/Big Datamedium1 min readPremium

Frequency

Low

Asked at 2 companies

Category

452

questions in Spark/Big Data

Difficulty Split

88E|81M|283H

in this category

Total Bank

1,863

across 7 categories

Asked at these companies

Deloitte Incedo

Interview Pro Tip

Red Flag: Watermark in ADF without idempotent sink—reruns can double-count. Pro-Move: Use Delta CDF + MERGE for true CDC; combine with streaming for near-real-time and batch for backfill in a unified pipeline.

Key Concepts Tested

Expert AnswerPremium

191 wordsInterview-ready

**Pattern**: Process only new/changed data by tracking last processed boundary (watermark) or using CDC. **ADF approach**: Watermark via lookup/stored procedure storing max(modified_date); filter source WHERE modified_date > @lastRun; parameterize pipeline for lastRun; write to sink. Use triggers for scheduling. **Databricks approach**: Delta Lake MERGE INTO for upserts; Change Data Feed (CDF) for CDC; read CDF or filter on _change_version....

The complete answer continues with detailed implementation patterns, architectural trade-offs, and production-grade considerations. It covers performance optimization strategies, common pitfalls to avoid, and real-world examples from companies like Deloitte, Incedo. The answer also includes follow-up discussion points that interviewers commonly explore.

Continue Reading the Full Answer

Unlock the complete expert answer with code examples, trade-offs, and pro tips - plus 1,863+ more.

Create Free Account - Unlock 30 Answers

Get PDF Bundle - from $21

Or upgrade to Platform Pro - $39

Engineers who used these answers got offers at

AmazonDatabricksSnowflakeGoogleMeta

Related Spark/Big Data Questions

mediumWhat is the difference between repartition and coalesce in Apache Spark?Free hardWhat is the difference between SparkSession and SparkContext in Spark?Free mediumWhat is the difference between cache() and persist() in Spark? When would you use each?Free mediumWhat is the difference between groupByKey and reduceByKey in Spark?Free mediumWhat is the difference between narrow and wide transformations in Apache Spark? Explain with examples.Free

According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 2 companies. DataEngPrep.tech maintains a curated database of 1,863+ real data engineering interview questions across 7 categories, verified by industry professionals.

← Back to all questions More Spark/Big Data questions →