Paytm Data Engineer Interview Questions (2026)

**Section 1 — The Context (The 'Why')** This system design addresses "Design the architecture for an Online Ticket Booking System..." at production scale. The primary challenges are throughput under variable load, fault tolerance across distributed components, and maintaining consistency guarantees that match business requirements. Naive monolithic approaches break when single points of failure emerge, network partitions occur, or backpressure from slow consumers cascades upstream. Data skew when hot keys overwhelm individual partitions, non-idempotent operations causing duplicate records on retry, and missing TTL policies leading to unbounded state growth are common failure modes that require explicit architectural choices. **Section 2 — The Diagram** ``` [Source] --> [Ingestion] --> [Processing] --> [Storage] --> [Serving] | | | | | v v v v v [Schema] [Buffer/Queue] [Compute] [Lake/DB] [API/BI] ``` **Section 3 — Component Logic** The ingestion layer buffers incoming data with backpressure handling to prevent overload when downstream processors cannot keep pace. Partitioning strategies determine parallelism and directly impact data skew mitigation; hot partitions require salting or secondary hashing to distribute load. The processing tier uses checkpointing and idempotent sinks to achieve exactly-once semantics on retries. Fan-out patterns allow one source to feed multiple consumers without re-ingestion. Storage applies TTL policies and partitioning for cost optimization and query performance. The serving layer exposes results with consistency guarantees appropriate to the use case. Exactly-once semantics require transactional commits and deduplication keys; at-least-once is acceptable only when idempotent sinks prevent duplicate side effects. Data skew mitigation via secondary key hashing prevents single partitions from becoming bottlenecks. In production, monitor consumer lag, checkpoint success rate, and sink write latency as primary SLOs. Partitioning strategies should align with query patterns; bucketing within partitions mitigates join skew. TTL policies on raw and intermediate data control storage cost while preserving replay capability for debugging and backfill. Data skew mitigation via salting or secondary hashing prevents single partitions from becoming bottlenecks. Exactly-once semantics require transactional commits at the sink; at-least-once delivery demands idempotent write logic to avoid duplicates. Fan-out patterns allow one source topic to feed multiple downstream consumers without re-ingestion. Backpressure handling ensures that slow processors do not cause unbounded buffer growth; Kafka consumer lag is a key metric. Schema evolution should follow additive-only rules where possible to avoid breaking consumer compatibility. The CAP trade-off should be documented per component: analytics typically favors AP, while financial reconciliation requires CP. Blast radius from component failure is bounded by replication and checkpointing; design for graceful degradation during partial outages. Cost optimization: use Spot instances for batch workloads and tier cold data to lower storage classes. Dead-letter queues preserve failed records for replay rather than dropping them. **Section 4 — The Trade-offs (The 'Senior' part)** - **CAP Theorem**: We choose AP (Availability + Partition Tolerance) for analytics pipelines where stale-by-minutes data is acceptable for dashboards, and we cannot afford downtime during partition events or consumer rebalances. For transactional or financial systems, CP (Consistency + Partition Tolerance) is preferred—we retry until success rather than serving incorrect data. The choice depends on whether the business can tolerate eventual consistency. - **Cost vs. Performance**: Managed services (Glue $0.44/DPU-hr, Kinesis, Lambda) vs self-managed (EMR $0.10/hr + EC2, Kafka on EC2) offer a clear trade-off: managed wins for operational simplicity and bursty workloads under 2 hours; self-managed wins for sustained 8hr+ daily jobs with approximately 60% cost savings. Storage tiering (S3 Standard to Glacier) and right-sized compute reduce ongoing cost. - **Blast Radius**: If the primary ingestion component fails, backpressure propagates upstream and producers may throttle. Processing failure triggers stage retry from the last checkpoint; at-most one checkpoint interval of reprocessing. Storage failure affects availability until replication promotes a new primary. The system self-heals via Kafka ISR for brokers, Spark task retry for executors, and database replication for persistence. Blast radius is typically bounded to a single partition group or job. **Section 5 — Pro-Tip** - **Pro-Move**: Design for failure from day one; use idempotent operations and checkpointing; partition by business keys for scalability. - **Red Flag**: Ignoring backpressure and exactly-once semantics leads to duplicates and data loss; always articulate checkpoint and dedup strategy.