Explain how Glue's Spark-based architecture handles data parallelism.

Section 1 — The Context (The 'Why')
AWS Glue runs Spark jobs on serverless DPUs—each DPU provides 4 vCPUs and 16GB RAM. The challenge is that Glue abstracts away cluster management, so engineers often treat it as a black box and miss parallelism tuning. Small file problems, incorrect partition counts, and wrong worker types cause jobs to run 5–10x slower than necessary. A naive approach uses default G.1X workers for memory-heavy jobs, leading to OOM or underutilization.

Section 2 — The Diagram

[S3 | RDS] --> [Glue Catalog]
   |
   v
[DPU Pool] 4vCPU/16GB each
   |
   +-> [Partition 1..N]
   |
   v
[Spark Executors] Write

Section 3 — Component Logic
Glue Catalog discovers schema and partitions from S3 or JDBC sources. The DPU is the parallelism unit—one DPU can run multiple Spark tasks. G.1X provides 1 DPU; G.2X provides 2 DPU for memory-intensive workloads. Partitions determine parallelism—each partition becomes a task. Glue job bookmarks enable incremental processing and idempotency by tracking the last processed file. Fan-out: one job can write to multiple targets. Data skew mitigation requires repartitioning or salting when source data is uneven. Overwrite by partition provides idempotent writes. Set spark.conf for partition count and parallelism.

Section 4 — The Trade-offs (The 'Senior' part)
CAP Theorem: Glue favors AP—serverless; jobs retry on failure; no persistent executor state. DPU allocation is eventually consistent. For ETL, we accept job rerun on failure (idempotent sink) over strong consistency.

Cost vs. Performance: Glue $0.44/DPU-hr. G.2X for memory-heavy jobs. 10 workers G.1X = $4.40/hr. EMR: $0.10 + ~$0.68/hr (m5.xlarge) = ~$0.78/node. Glue wins for jobs under 2 hours; EMR wins for 8hr+ daily.

Blast Radius: Glue job fail: automatic retry. DPU preemption: job restarts. Catalog fail: cannot discover tables. Bookmark: resumes from last processed file.

Section 5 — Pro-Tip
Pro-Move: DPU = 4vCPU; use G.2X for memory; job bookmarks for incremental; tune spark.conf for partitions.
Red Flag: Treating Glue as black box without parallelism tuning—leaves significant cost and latency on the table. From a Principal Engineer perspective, the key differentiators are operational rigor—defined SLAs, runbooks, and chaos testing—and cost consciousness—right-sizing, reserved capacity, and incremental processing to minimize compute. The failure modes we guard against include partition events (Kafka ISR, consumer rebalance), poison messages (DLQ with alerting), and offset loss (S3 checkpoint). Interview red flags include missing idempotency (duplicates on retry), no DLQ (one bad record blocks the pipeline), and checkpointing to ephemeral storage (state lost on preemption). Production systems require monitoring of consumer lag, data freshness SLOs, and cost per record processed. Schema evolution should be additive-only with Schema Registry; partitioning strategies must align with query filters (date, region); blast radius is contained through replication, circuit breakers, and graceful degradation. When choosing between CP and AP: ledger and warehouse layers favor consistency; streams and caches favor availability. Cost optimization: Glue for bursty jobs under 2 hours; EMR for sustained 8+ hour workloads. Always quantify improvements—latency reduction, cost savings, volume handled. Data skew mitigation via salting and AQE prevents hotspot tasks; exactly-once semantics require idempotent sinks; fan-out patterns enable multiple consumers without duplication. TTL policies on Bronze reduce storage cost; incremental processing cuts compute by 90% versus full scans. Replication factor of three with min.insync.replicas=2 ensures durability; consumer count should match or exceed partition count; event-time over processing-time handles late arrivals correctly. Medallion architecture separates raw from curated; quality gates at Silver prevent bad data propagation; conformed dimensions enable cross-mart consistency. In interviews, demonstrate production experience by citing specific metrics: P95 latency, cost per million events, recovery time objective. Avoid generic answers; tie each design choice to a measurable outcome. The trade-off between consistency and availability is per-component: choose CP for financial transactions, AP for analytics. Scale testing should cover 10x peak load; runbooks should document failure recovery steps. Blue-green deployments enable zero-downtime schema evolution; view abstraction with COALESCE supports additive column migration. For real-time systems, define SLOs before building—lag under five minutes and freshness under one hour are common targets. Correlation IDs in log records enable end-to-end tracing when debugging production incidents. Reserve capacity for traffic spikes; implement circuit breakers to prevent cascading failures across dependent services. Document design decisions and their trade-offs for future maintainability. This demonstrates production-grade system design thinking.