Design a data pipeline to ingest and process clickstream data in near real-time

Section 1 — The Context (The 'Why')
Clickstream pipelines process high-volume, low-latency events from web and mobile. The primary challenge is handling millions of events per second with sub-second aggregation for sessionization, funnel analysis, and A/B test attribution. Hot keys (popular pages, viral campaigns) cause severe data skew; out-of-order events from mobile networks complicate session boundaries.

Section 2 — The Diagram

[SDK/Beacon] --> [CloudFront] --> [Kinesis/Kafka] --> [Flink Sessionizer]
     |                   |                |                    |
     v                   v                v                    v
[Event Schema]      [Edge Logs]    [Partition by       [Redis/ClickHouse]
                                   user_id]            [Funnel/Analytics]

Section 3 — Component Logic
Events are beaconed or sent via SDK to an edge layer. Kinesis or Kafka ingests with partitioning by user_id or session_id to preserve ordering per user; data skew mitigation uses secondary hashing for hot keys. Flink performs sessionization with idle gaps (e.g., 30 min) and event-time watermarks. Exactly-once semantics via Kafka transactional producer and idempotent writes prevent double-counting. TTL policies on session state prevent unbounded memory. Fan-out: one stream feeds session, attribution, and raw archive consumers. In production, monitor consumer lag, checkpoint success rate, and sink write latency as primary SLOs. Partitioning strategies should align with query patterns; bucketing within partitions mitigates join skew. TTL policies on raw and intermediate data control storage cost while preserving replay capability for debugging and backfill. Data skew mitigation via salting or secondary hashing prevents single partitions from becoming bottlenecks. Exactly-once semantics require transactional commits at the sink; at-least-once delivery demands idempotent write logic to avoid duplicates. Fan-out patterns allow one source topic to feed multiple downstream consumers without re-ingestion. Backpressure handling ensures that slow processors do not cause unbounded buffer growth; Kafka consumer lag is a key metric. Schema evolution should follow additive-only rules where possible to avoid breaking consumer compatibility. The CAP trade-off should be documented per component: analytics typically favors AP, while financial reconciliation requires CP. Blast radius from component failure is bounded by replication and checkpointing; design for graceful degradation during partial outages. Cost optimization: use Spot instances for batch workloads and tier cold data to lower storage classes. Dead-letter queues preserve failed records for replay rather than dropping them.

Section 4 — The Trade-offs (The 'Senior' part)