Explain Azure Databricks architecture and its integration with other Azure services.

Section 1 — The Context (The 'Why')
Azure Databricks runs customer workloads in the customer's own VPC while the control plane (workspace, jobs, UI) resides in Databricks cloud. This separation creates operational complexity: data must never leave the customer tenant, while job orchestration and metadata live elsewhere. A naive assumption that Databricks hosts data leads to compliance failures; multi-service integration (ADLS, Event Hubs, Synapse) requires careful networking and identity configuration.

Section 2 — The Diagram

[Azure AD] --> [Databricks Workspace]
   |
   +-> [ADLS Gen2] Data
   +-> [Data Factory] Orch
   +-> [Event Hubs] Stream
   +-> [Synapse] DW

Section 3 — Component Logic
Databricks Workspace is the control plane entry point—notebooks, jobs, and clusters are managed here. The workspace never stores customer data. ADLS Gen2 holds all data in the customer subscription; Databricks clusters attach via managed identity and process data in-place. Data Factory orchestrates pipeline runs and triggers Databricks jobs. Event Hubs streams events into Databricks for real-time processing. Synapse can share the same ADLS data via external tables—avoid data duplication. Idempotency is achieved via Delta merge. Fan-out from Event Hubs allows multiple consumers. TTL policies on ADLS lifecycle manage retention. Unity Catalog provides governance across Databricks and Synapse.

Section 4 — The Trade-offs (The 'Senior' part)
CAP Theorem: Databricks on Azure chooses AP for data processing—clusters are stateless; data in ADLS is the source of truth. Workspace and job metadata in the control plane favor CP. For data processing, eventual consistency across cluster restarts is acceptable.

Cost vs. Performance: Databricks $0.40–0.75/DBU vs Azure Synapse $1.23/hr dedicated. ADLS $0.018/GB. Event Hubs $0.028/million events. Databricks + ADLS cheaper for variable workloads; Synapse for sustained DW. Unity Catalog adds ~$0.10/DBU.

Blast Radius: Control plane out: Cannot create jobs; existing clusters continue running. Data plane remains in customer VPC. Cluster fail: restart; data in ADLS is safe. Workspace fail: use REST API for automation.

Section 5 — Pro-Tip
Pro-Move: Keep data plane in your VPC; use ADLS as source of truth; VNet injection; Unity Catalog for governance; ADF for orchestration.
Red Flag: Assuming Databricks hosts your data—data stays in your cloud; control plane is remote. From a Principal Engineer perspective, the key differentiators are operational rigor—defined SLAs, runbooks, and chaos testing—and cost consciousness—right-sizing, reserved capacity, and incremental processing to minimize compute. The failure modes we guard against include partition events (Kafka ISR, consumer rebalance), poison messages (DLQ with alerting), and offset loss (S3 checkpoint). Interview red flags include missing idempotency (duplicates on retry), no DLQ (one bad record blocks the pipeline), and checkpointing to ephemeral storage (state lost on preemption). Production systems require monitoring of consumer lag, data freshness SLOs, and cost per record processed. Schema evolution should be additive-only with Schema Registry; partitioning strategies must align with query filters (date, region); blast radius is contained through replication, circuit breakers, and graceful degradation. When choosing between CP and AP: ledger and warehouse layers favor consistency; streams and caches favor availability. Cost optimization: Glue for bursty jobs under 2 hours; EMR for sustained 8+ hour workloads. Always quantify improvements—latency reduction, cost savings, volume handled. Data skew mitigation via salting and AQE prevents hotspot tasks; exactly-once semantics require idempotent sinks; fan-out patterns enable multiple consumers without duplication. TTL policies on Bronze reduce storage cost; incremental processing cuts compute by 90% versus full scans. Replication factor of three with min.insync.replicas=2 ensures durability; consumer count should match or exceed partition count; event-time over processing-time handles late arrivals correctly. Medallion architecture separates raw from curated; quality gates at Silver prevent bad data propagation; conformed dimensions enable cross-mart consistency. In interviews, demonstrate production experience by citing specific metrics: P95 latency, cost per million events, recovery time objective. Avoid generic answers; tie each design choice to a measurable outcome. The trade-off between consistency and availability is per-component: choose CP for financial transactions, AP for analytics. Scale testing should cover 10x peak load; runbooks should document failure recovery steps. Blue-green deployments enable zero-downtime schema evolution; view abstraction with COALESCE supports additive column migration. For real-time systems, define SLOs before building—lag under five minutes and freshness under one hour are common targets. Correlation IDs in log records enable end-to-end tracing when debugging production incidents. Reserve capacity for traffic spikes; implement circuit breakers to prevent cascading failures across dependent services. Document design decisions and their trade-offs for future maintainability. This demonstrates production-grade system design thinking.