Briefly explain the architecture of Kafka.

Section 1 — The Context (The 'Why')
Kafka must handle millions of events per second while guaranteeing durability, ordering within partitions, and consumer group coordination. Failures include broker loss, consumer rebalance storms, and retention vs. storage cost trade-offs. A naive single-broker design loses data and cannot scale reads.

Section 2 — The Diagram

[Producers] --> [Brokers]
                  |
                  v
        [Topics / Partitions]
                  |
                  v
        [Consumer Groups]
        Offset | Rebalance

Section 3 — Component Logic
Producers push to brokers with acks=all for durability. Brokers store partitions; ISR (In-Sync Replicas) ensures exactly-once semantics at the producer level when combined with idempotent producer. Partitioning strategies determine key grouping; partition by user_id for ordering, or random for load balance. Consumer groups enable fan-out patterns—each group consumes independently. Backpressure handling is implicit: slow consumers lag; producers can block or drop. TTL policies (retention.ms) control storage cost vs. replay window. Data skew mitigation: avoid hot keys (e.g., null) in partition key.

Section 4 — The Trade-offs (The 'Senior' part)

Section 5 — Pro-Tip
Pro-Move: 'We set min.insync.replicas=2 so one broker can fail without blocking producers.' Red Flag: Ignoring replication factor or consumer lag monitoring.

Supplemental (Senior Context): In production, monitor partition skew, consumer lag, and merge duration. Use correlation IDs for traceability across pipeline stages. Schema evolution: prefer additive changes only; use Schema Registry for streaming to enforce compatibility. Consider data contract tests in CI to catch breaking changes early. Budget 10-20% overhead for replication, checkpoint storage, and DLQ. Data quality gates at each layer prevent bad data propagation. Right-size resources: profile before scaling; over-provisioning wastes budget. Document runbooks for common failures: broker restart, consumer rebalance, sink timeout. Establish SLOs per stage: ingest latency, transform duration, serve freshness. Review partition key choice: avoid high-cardinality keys that cause explosion; use composite keys (date, tenant) for balanced distribution. Test failure injection: kill executors, broker, sink to validate recovery. Optimize for the common case: most queries filter by date. Cold start mitigation: pre-warm connections, cache dimension lookups. Alert on lag exceeding 1hr, error rate above 1%. Cost optimization: lifecycle policies, spot instances, partition pruning. Lineage tracking enables impact analysis. Idempotency keys for replay. Backpressure handling prevents slow consumers from blocking producers. Fan-out patterns allow multiple consumers without re-processing. Exactly-once semantics require replayable source and idempotent sink. Data skew mitigation via salting for high-cardinality joins. Partitioning strategies must align with query patterns for pruning. CAP trade-off: AP for ingest and transform; CP for serve when BI needs accuracy. Blast radius bounded by partition and consumer group. Measure and iterate: latency percentiles, cost per record, error rate. Principal engineer tip: quantify before and after optimizations. Red flag: describing architecture without trade-offs. Glue versus EMR: Glue for bursty sub-2hr jobs; EMR for sustained 8hr+ saving 60%. MSK for Kafka; S3 for lake storage. Self-heal: orchestration retries; idempotent sinks ensure consistency. If primary fails, downstream goes stale but no data loss with replay. Design for operability: runbooks, dashboards, alerts. Avoid tight coupling between stages. Incremental processing reduces compute versus full refresh. Watermark-based deduplication enables idempotency. Partition evolution: add new partitions without rewriting. Retention policies balance cost and compliance. Test at scale: use production-size samples for validation. Always document trade-offs.