How do you handle exceptions in data ingestion?

Section 1 — The Context (The 'Why')
Exception handling in data ingestion determines whether the system fails safely or corrupts silently. Unbounded retries can DDoS sources (rate limits, IP bans); silent drops lose data forever; and missing circuit breakers allow cascading failures. At Gartner-scale advisory, heterogeneous sources (APIs, DBs, files) have wildly different failure modes: transient network blips vs permanent schema breaks. A naive approach—retry forever or drop on error—either overwhelms upstream or creates undetectable data loss. Key failure modes: connector OOM under backpressure, schema drift causing validation storms, and DLQ overflow when bad data volume spikes.

Section 2 — The Diagram

[Source]---->[Connector]---->[Validate]
   |              |               |
   v              v               v
[Retry+Circuit]  [OK]        [Fail->DLQ]
   |              |               |
   +--------------+----------->[Alert]

Section 3 — Component Logic
The connector pulls data with retry logic (exponential backoff, capped attempts—e.g., 5 retries with 2^n seconds) and a circuit breaker that opens after N consecutive failures, failing fast to protect the source. Validation (schema, format, business rules) separates good from bad; good records flow to the sink; bad records follow a fan-out pattern to the DLQ with full error context (record, error message, timestamp). The DLQ enables replay after fixing parsers. Idempotency in the sink ensures safe retries without duplicates. Backpressure handling: when the consumer lags, the connector slows or blocks rather than buffering unboundedly. Alerts fire on DLQ growth, circuit open, or validation failure rate spikes. Implement a batch job to periodically drain and reprocess the DLQ; tag records with error type for triage. Rate-limit retries per source to avoid hammering failing APIs.

Section 4 — The Trade-offs (The 'Senior' part)

Design principles: Cap retries at 5–10; use exponential backoff (1s, 2s, 4s, 8s) to avoid thundering herd. Open circuit after 5 consecutive failures; half-open after 30s to test recovery. Always include record ID, timestamp, and error message in DLQ payload for debugging. Distinguish transient errors (retry) from permanent errors (DLQ, manual fix); use error classification to route appropriately.

Section 5 — Pro-Tip