Question 1

What architecture are you following in your current project, and why?

Accepted Answer

**Section 1 — The Context (The 'Why')**
The primary challenge in choosing and justifying an architecture is alignment between technical complexity, operational cost, and business latency requirements. A naive monolithic pipeline fails when schema evolution hits, when a single component becomes a bottleneck, or when teams scale—causing coordination overhead and deployment risk....

Question 2

Briefly introduce yourself and walk us through your journey as a Data Engineer so far.

Accepted Answer

**Situation**: I joined as a software engineer and saw data as a bottleneck—pipelines broke, nobody trusted the numbers. **Task**: Transition into data engineering and build reliable, scalable systems. **Action**: I moved from ETL dev to owning cloud data platforms—designed data lakes on AWS/GCP, optimized Spark jobs (reduced costs 40% via partition pruning and skew fixes), implemented Kafka/Flink streaming, and led migrations to Delta Lake....

Question 3

What are the key components of AWS Glue, and how do they work together?

Accepted Answer

Glue Catalog: Central metadata store (Hive-compatible)—enables querying S3 data via Athena/Redshift Spectrum without moving it. Glue Crawlers: Schema discovery and Catalog population—useful for ad-hoc sources; at scale, prefer schema-as-code to avoid crawler cost and drift. Glue ETL Jobs: Serverless Spark for transforms; auto-scaling, pay-per-DPU. Glue DataBrew: Visual prep for non-engineers. Glue Schema Registry: Schema evolution for streaming (Kafka, Kinesis)....

Question 4

Have you worked on Data Warehousing projects?

Accepted Answer

**Architectural context**: A data warehouse is the semantic layer between raw data and business decisions. Design choices—star vs snowflake, SCD strategy, partitioning—directly impact query latency, storage cost, and maintenance burden. **Key responsibilities**: (1) **Schema design**: Star for BI simplicity, snowflake for normalized flexibility. SCD Type 2 for slowly changing dimensions (audit trail, point-in-time correctness)....

Question 5

How would you read data from a web API? What steps would you follow after reading the data?

Accepted Answer

**Why this matters**: APIs are external dependencies—unreliable, rate-limited, schema-evolving. Production ingestion must be resilient, auditable, and idempotent. **Steps**: (1) **Contract & auth**: Define schema (Pydantic, JSON Schema); store credentials in secrets manager (Vault, AWS Secrets). OAuth tokens need refresh logic. (2) **Pagination & rate limiting**: Offset/cursor-based; loop until empty. Respect Retry-After; implement exponential backoff....

Question 6

What is the difference between OLTP and OLAP?

Accepted Answer

**Why the distinction exists**: They serve different access patterns. OLTP = many small, random writes and point reads. OLAP = few, large sequential scans and aggregations. Optimizing for one degrades the other. **OLTP**: Row-oriented storage (fast single-row access). Normalized schema (3NF) to avoid update anomalies. Indexes for lookup (B-tree). ACID for consistency. High concurrency via locking/mvcc. Examples: PostgreSQL, MySQL, Oracle....

Question 7

What is normalization and denormalization? When would you use each?

Accepted Answer

**Architectural Logic**: Normalization (1NF→3NF): Remove redundancy, enforce integrity. OLTP optimized for writes, consistency. Denormalization: Add redundancy for read performance. OLAP/data marts optimized for analytics. **Why**: Normalized = single source of truth, easier updates. Denormalized = fewer joins, faster queries, larger storage. **Scalability**: Normalized scales for writes; denormalized scales for reads but requires ETL to maintain....

Question 8

When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.

Accepted Answer

DataFrame is an untyped collection of Row objects with schema at runtime; Dataset[T] is typed with compile-time safety. In Scala, DataFrame = Dataset[Row]. Architectural why: Dataset enables domain modeling (e.g., Dataset[Order])—catch errors at compile time, better IDE support, and Catalyst can optimize typed encoders. Scalability: both use Tungsten and Catalyst; Dataset adds encoder overhead but marginal for most workloads. Portability trade-off: PySpark has only DataFrame—no typed Dataset....

Question 9

How do you handle memory management in Python?

Accepted Answer

**Mechanisms**: (1) Reference counting—object freed when refcount=0; (2) Cycle detector (gc)—handles circular refs; (3) PyMalloc—small object pools. **Manual tools**: `del` to drop refs; `gc.collect()` to force cycle detection; `gc.set_threshold()` to tune. **Why it matters**: Long-running services (ETL, APIs) can leak via cycles or unclosed resources. **Scalability trade-off**: Large heaps increase GC pause; generators/iterators reduce peak memory....

Question 10

Briefly introduce yourself and walk us through your journey as a Data Engineer so far.

Accepted Answer

**Situation**: I started in software engineering, moved into ETL, and evolved toward platform-scale data engineering. **Task**: Build reliable, scalable pipelines that serve both analysts and ML teams while managing cost and complexity. **Action**: I've led cloud data lake builds (AWS, GCP), optimized Spark jobs for billion-event daily volumes, implemented streaming with Kafka and Flink, and driven migrations from legacy warehouses....

ETL Interview Questions

Companies asking these questions

All 60 Questions

More Interview Prep Guides

Get All Answers in PDF Format

ETL Interview Questions

Companies asking these questions

All 60 Questions

More Interview Prep Guides

Get All Answers in PDF Format