How would you schedule a recurring pipeline in Data Fusion?

Section 1 — The Context (The 'Why')
The primary challenge in 'How would you schedule a recurring pipeline in Data Fusion?' centers on designing for production scale, correctness guarantees, and operational resilience. A naive or underspecified design fails under load: single points of failure cascade, non-idempotent operations cause duplicates on retry, and lack of observability blocks root-cause analysis. At enterprise scale, failure modes multiply—what works for small batches breaks when volume grows 10x. The diagram above shows the key components; each must be chosen for its role in ensuring backpressure handling (protecting sources when consumers lag), idempotency (safe retries), and partitioning strategies (horizontal scale). Senior architects prioritize explicit trade-offs: CAP choices, cost vs. latency, and blast radius containment.

Section 2 — The Diagram

[Data Fusion]-->[Studio]
   |
   v
[Pipeline Design]
   |
   v
[Configure Schedule]
   |
   v
[Cron: 0 2   *]
   |
   v
[Dataproc Profile|Runs]

Section 3 — Component Logic
Each component in the diagram above serves a critical role. The architecture must apply backpressure handling to prevent overwhelming sources when consumers lag—rate limits and flow control propagate upstream so producers slow rather than overflow buffers. Idempotency at the sink ensures safe retries without duplicates; use deterministic keys (e.g., hash of business key + timestamp) so replay produces the same result. Partitioning strategies (by date, region, or business key) enable parallel processing and cost-efficient query pruning; undersizing partitions causes hot spots and data skew. For streaming, exactly-once semantics require checkpointing plus transactional sinks (Kafka + Delta MERGE); without both, duplicates or gaps occur. Fan-out patterns allow one source to feed multiple consumers independently; each consumer can scale and fail without blocking others. TTL policies control retention and lifecycle costs—raw zones kept short for replay, curated zones longer. Data skew mitigation (salting hot keys, broadcast joins for small dimensions) prevents stragglers from dominating runtime; one skewed partition can 10x job duration. For this specific design: Cron scheduling. Parameterized dates. Idempotency for reruns. Implementation choices depend on throughput, latency SLA, and compliance: high-throughput batch favors Spark/EMR; sub-second streaming needs Flink; warehouse loads prefer dbt or merge-based loads. Monitor partition lag, validation failure rates, and sink latency; alert on drift. When designing from scratch, prefer managed services (Kinesis, Glue, Athena) for faster iteration; migrate to self-managed (Kafka, EMR) when cost or control dictates. Always document assumptions (e.g., max late arrival, retention window) so future changes are informed. Test failure injection (kill workers, delay sources) to validate recovery behavior before production. This discipline separates production-grade systems from proof-of-concepts.

Section 4 — The Trade-offs (The 'Senior' part)