Explain the trade-offs between batch and real-time data processing. Provide examples of when each is appropriate.

Why the distinction matters: Systems are optimized for different access patterns. Batch assumes bounded, complete datasets; streaming assumes unbounded, infinite data. The choice cascades into storage layout, compute model, operational complexity, and cost structure. Batch: Designed for high-throughput, bounded processing. Lower cost per record due to amortized compute (e.g., spot instances). Easier to reason about correctness (all-or-nothing). Trade-off: End-to-end latency is O(hours). Use for: nightly reconciliation, ML training, complex multi-table joins, regulatory reporting where freshness isn't critical. Streaming: Designed for low-latency, unbounded processing. Higher cost (always-on consumers, stateful infra). Requires explicit semantics (at-least-once vs exactly-once), checkpointing, backpressure, and schema evolution. Use for: fraud detection (<100ms), inventory updates, recommendation systems, alerting. Architectural trade-off: Hybrid (Lambda/Kappa) is common—streaming for real-time views, batch for historical backfill and complex aggregations. Cost implication: A streaming pipeline running 24/7 can cost 3–5x a daily batch job for the same volume; justify with business value of latency.