How would you implement custom alarms for data delays or job failures?

Implementing custom alarms requires an architectural view of observability layers. WHY: Silent failures erode trust; SLA breaches have regulatory and revenue impact. Design a multi-tier stack: (1) COLLECTION—instrument jobs with completion timestamps, data freshness (max(event_ts) per partition), failure counts via CloudWatch/Datadog/Prometheus; (2) THRESHOLD ENGINE—define SLAs (e.g., daily batch by 6 AM) and trigger PagerDuty/SNS; (3) DEPENDENCY CHECKS—upstream sensors block downstream runs. SCALABILITY: Use idempotent alerting (dedupe by incident key) to avoid alert storms; aggregate per partition for large tables to reduce metric cardinality. COST: Alert volume scales with job count—consolidate rules, use severity-based escalation; runbooks reduce MTTR and avoid unnecessary escalations. Production: Airflow DAG with sensor tasks checking S3 for new partitions; Slack webhook + PagerDuty with escalation policies; Grafana as single pane.