Conceptualize and design a real-time streaming data pipeline end-to-end.

Section 1 — The Context (The 'Why')
Real-time streaming pipelines face a fundamental tension: events arrive continuously at high velocity while downstream consumers demand low latency, yet the system must guarantee no data loss during broker failures or consumer restarts. A naive approach—writing directly to a database per event—collapses under load; checkpointing to local disk loses state on executor preemption. The primary challenge is achieving exactly-once semantics across a distributed, fault-prone pipeline while maintaining sub-minute latency SLAs.

Section 2 — The Diagram

[Events] --> [Kinesis/Kafka]
      |
      v
[Spark/Flink] --> [Delta Lake]
      |
      v
[Bronze] --> [Silver] --> [Gold]
      |
      v
[BI | ML | Dashboards]

Section 3 — Component Logic
The Kinesis/Kafka tier acts as the durable ingestion buffer—events land here first regardless of downstream processing speed, enabling backpressure handling at the source. We choose a partitioned stream because parallelism equals throughput; partitioning by business key preserves order within a partition. Spark or Flink executes micro-batches or continuous processing; we choose Spark Structured Streaming for SQL familiarity and Glue integration, or Flink for true event-time windowing and lower latency. The processor must implement exactly-once semantics via idempotent sinks—a Delta Lake merge on (primary_key, batch_id) ensures duplicate events from retries produce one output row. Bronze/Silver/Gold implements the Medallion architecture: Bronze is raw append-only; Silver applies deduplication and merge; Gold contains business aggregates. Data skew mitigation via salting on high-cardinality join keys prevents hotspot partitions. TTL policies on Bronze move cold data to Glacier after retention windows.

Section 4 — The Trade-offs (The 'Senior' part)
CAP Theorem: We choose AP because for this analytics pipeline, stale-by-five-minutes data is acceptable for dashboards, and we cannot afford downtime during partition events or consumer rebalances. Exactly-once semantics at the sink provide transactional consistency within the pipeline; the stream layer trades strong C for availability.

Cost vs. Performance: Kinesis Data Streams ($0.015/shard-hr + $0.014/GB) vs MSK ($0.21/broker-hr): Kinesis wins for AWS-native simplicity and lower ops. Glue Streaming ($0.44/DPU-hr) vs EMR ($0.10/hr + EC2): Glue wins for bursty jobs under 2 hours; EMR saves ~60% for sustained 8hr+ workloads.

Blast Radius: If a Kafka broker fails: ISR promotes a new leader within 10 seconds. Consumers rebalance within 30 seconds. No data loss with RF=3 and min.insync.replicas=2. Blast radius is bounded to one partition group—other partitions continue serving traffic.

Section 5 — Pro-Tip
Pro-Move: Define a 5-min P95 latency SLA; checkpoint to S3 not local disk; use idempotent Delta merge for exactly-once; set partition count = 2x consumer count for headroom.
Red Flag: No latency SLA and checkpointing to local disk—state lost on executor preemption; interviewers will probe for production gaps. From a Principal Engineer perspective, the key differentiators are operational rigor—defined SLAs, runbooks, and chaos testing—and cost consciousness—right-sizing, reserved capacity, and incremental processing to minimize compute. The failure modes we guard against include partition events (Kafka ISR, consumer rebalance), poison messages (DLQ with alerting), and offset loss (S3 checkpoint). Interview red flags include missing idempotency (duplicates on retry), no DLQ (one bad record blocks the pipeline), and checkpointing to ephemeral storage (state lost on preemption). Production systems require monitoring of consumer lag, data freshness SLOs, and cost per record processed. Schema evolution should be additive-only with Schema Registry; partitioning strategies must align with query filters (date, region); blast radius is contained through replication, circuit breakers, and graceful degradation. When choosing between CP and AP: ledger and warehouse layers favor consistency; streams and caches favor availability. Cost optimization: Glue for bursty jobs under 2 hours; EMR for sustained 8+ hour workloads. Always quantify improvements—latency reduction, cost savings, volume handled. Data skew mitigation via salting and AQE prevents hotspot tasks; exactly-once semantics require idempotent sinks; fan-out patterns enable multiple consumers without duplication. TTL policies on Bronze reduce storage cost; incremental processing cuts compute by 90% versus full scans. Replication factor of three with min.insync.replicas=2 ensures durability; consumer count should match or exceed partition count; event-time over processing-time handles late arrivals correctly. Medallion architecture separates raw from curated; quality gates at Silver prevent bad data propagation; conformed dimensions enable cross-mart consistency. In interviews, demonstrate production experience by citing specific metrics: P95 latency, cost per million events, recovery time objective. Avoid generic answers; tie each design choice to a measurable outcome. The trade-off between consistency and availability is per-component: choose CP for financial transactions, AP for analytics. Scale testing should cover 10x peak load; runbooks should document failure recovery steps. Blue-green deployments enable zero-downtime schema evolution; view abstraction with COALESCE supports additive column migration. For real-time systems, define SLOs before building—lag under five minutes and freshness under one hour are common targets. Correlation IDs in log records enable end-to-end tracing when debugging production incidents. Reserve capacity for traffic spikes; implement circuit breakers to prevent cascading failures across dependent services. Document design decisions and their trade-offs for future maintainability. This demonstrates production-grade system design thinking.