Walk through a production incident where data freshness or correctness was at risk. How did you balance immediate mitigation vs. root-cause remediation? What architectural changes would prevent recurrence, and what are the cost vs. reliability trade-offs?

Situation: Pipeline failed at 2 AM; source schema change (new required column) broke ingestion. Mitigation vs. Remediation: Quick fix (default column, redeploy) restores service; proper fix (schema validation, evolution policy) prevents recurrence. Architectural Logic: Schema-on-write pipelines fail on evolution; schema validation (e.g., Glue Schema Registry, Avro) catches drift early. Resiliency vs. Cost: Retries, dead-letter queues, idempotency add complexity; schema validation adds latency and ops. Why prioritize: One outage can cost more than months of validation logic. Scalability: As sources grow, manual handling doesn't scale; automated schema evolution and alerting are essential. Best practice: Runbooks, monitoring, on-call rotation, blameless post-incident review. Document: what failed, why, and what prevents it.