Design a data pipeline for streaming analytics.

Section 1 — The Context (The 'Why')
Streaming analytics pipelines face the fundamental tension between low-latency ingestion and consistent aggregation at scale. The primary challenge is maintaining exactly-once or at-least-once semantics while supporting late-arriving events, handling backpressure when downstream processors cannot keep pace, and avoiding data skew when hot partitions (e.g., popular products) overwhelm single workers. A naive fire-and-forget approach breaks under burst traffic: consumers fall behind, watermarks stall, and dashboards show stale or incorrect metrics.

Section 2 — The Diagram

[API/Events] --> [Kafka Cluster] --> [Flink/Spark] --> [Delta/Redis]
     |                  |                  |               |
     v                  v                  v               v
[Schema Reg]       [Partitioned    [State Store]     [BI Dashboard]
                   Buffered]       [Watermarks]

Section 3 — Component Logic
The event ingestion layer (API gateway, mobile SDKs) publishes raw events to Kafka. Kafka serves as a durable, partition-tolerant buffer with configurable replication factor; its partitioning strategy (e.g., by user_id or session_id) directly impacts downstream parallelism and data skew. The stream processor (Flink or Spark Structured Streaming) consumes from Kafka with exactly-once semantics via transactional commits and checkpointing. Windowed aggregation uses event-time watermarks to handle late arrivals; a TTL policy on state stores prevents unbounded memory growth. The serving layer (Delta Lake for batch access, Redis for sub-second dashboards) enables both historical querying and real-time read-through. Backpressure handling flows upstream: when Flink cannot process fast enough, Kafka consumer lag increases and producers may throttle or buffer. In production, monitor consumer lag, checkpoint success rate, and sink write latency as primary SLOs. Partitioning strategies should align with query patterns; bucketing within partitions mitigates join skew. TTL policies on raw and intermediate data control storage cost while preserving replay capability for debugging and backfill. Data skew mitigation via salting or secondary hashing prevents single partitions from becoming bottlenecks. Exactly-once semantics require transactional commits at the sink; at-least-once delivery demands idempotent write logic to avoid duplicates. Fan-out patterns allow one source topic to feed multiple downstream consumers without re-ingestion. Backpressure handling ensures that slow processors do not cause unbounded buffer growth; Kafka consumer lag is a key metric. Schema evolution should follow additive-only rules where possible to avoid breaking consumer compatibility. The CAP trade-off should be documented per component: analytics typically favors AP, while financial reconciliation requires CP. Blast radius from component failure is bounded by replication and checkpointing; design for graceful degradation during partial outages. Cost optimization: use Spot instances for batch workloads and tier cold data to lower storage classes. Dead-letter queues preserve failed records for replay rather than dropping them.

Section 4 — The Trade-offs (The 'Senior' part)