Question 1

What are Airflow Operators? Give examples.

Accepted Answer

Airflow Operators define a single unit of work in a DAG—each operator performs one atomic, idempotent task. **Why they matter**: They encapsulate work so DAGs remain declarative and schedulable; the scheduler doesn't need to understand task logic. **Examples**: BashOperator, PythonOperator, SqlOperator, HTTPOperator, DockerOperator, KubernetesPodOperator, Sensor. **Scalability**: Heavy logic should live in external scripts or services; operators should only orchestrate....

Question 2

How do you remove duplicate rows in BigQuery?

Accepted Answer

Approach: Use ROW_NUMBER() OVER (PARTITION BY dedup_keys ORDER BY tie_breaker) to define which row to keep; filter rn = 1. Preferred pattern: CREATE OR REPLACE TABLE ... AS SELECT * EXCEPT(rn) FROM (SELECT *, ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1. Why CREATE OR REPLACE over DELETE: BigQuery is columnar; DELETE is a rewrite under the hood. For large tables, CREATE OR REPLACE is a single scan+write vs DELETE's read-modify-write....

Question 3

What are the key components of AWS Glue, and how do they work together?

Accepted Answer

Glue Catalog: Central metadata store (Hive-compatible)—enables querying S3 data via Athena/Redshift Spectrum without moving it. Glue Crawlers: Schema discovery and Catalog population—useful for ad-hoc sources; at scale, prefer schema-as-code to avoid crawler cost and drift. Glue ETL Jobs: Serverless Spark for transforms; auto-scaling, pay-per-DPU. Glue DataBrew: Visual prep for non-engineers. Glue Schema Registry: Schema evolution for streaming (Kafka, Kinesis)....

Question 4

What is Azure Data Factory (ADF), and what are its main components?

Accepted Answer

ADF is a cloud-native data integration service for orchestration and movement. Components: Pipelines (logical groups of activities), Activities (Copy, Lookup, Databricks, Data Flow), Datasets (structure definitions), Linked Services (connection configs), Triggers (schedule or event-based), Integration Runtime (IR—compute for execution). Flow: Linked Service -> Dataset -> Activity -> Pipeline -> Trigger....

Question 5

What is Snowflake's architecture, and why is it unique?

Accepted Answer

**Section 1 — The Context (The 'Why')**
Traditional data warehouses collapse under elastic concurrency: fixed clusters either over-provision (cost) or under-provision (queuing). Storage-compute coupling means scaling queries requires scaling storage nodes....

Question 6

What is the difference between S3 and HDFS?

Accepted Answer

S3: Object storage—durable, highly available, decoupled from compute. Pay-per-use, virtually unlimited scale. Eventual consistency (now strong for new overwrites). No data locality. HDFS: Distributed file system—block-based, colocated with compute nodes. Data locality reduces network I/O; strong consistency. Requires cluster management. Why it matters: S3 enables cloud-native, serverless patterns (Lambda, Glue, Athena); HDFS optimizes for batch processing where locality matters....

Question 7

What is the role of AWS Lambda in a data engineering pipeline?

Accepted Answer

Lambda is serverless compute for event-driven, short-lived tasks. Roles: (1) Orchestration: Trigger Glue, EMR, or Step Functions when files land (S3), events arrive (Kinesis), or schedules fire. (2) Light transforms: JSON→Parquet, small aggregations, API calls. (3) Alerts: Validate data, send Slack/PagerDuty on anomalies. Why: No server management; scale to thousands of concurrent invocations; pay per request and duration. Limitations: 15-min timeout, 10GB memory, 6MB payload (sync)....

Question 8

What is the role of the Integration Runtime (IR) in ADF?

Accepted Answer

IR is the compute layer that executes ADF activities—data movement, transformations, and external calls. Types: (1) Azure IR: Managed, cloud-native—for cloud-to-cloud copies, Data Flows, Databricks calls. No maintenance. (2) Self-hosted IR: Install on your VM or container—for on-prem, VNet, or private data sources. You manage scaling and uptime. (3) Azure-SSIS IR: For running SSIS packages in Azure. Why it matters: Data never flows through ADF; it flows through IR. Network topology (cloud vs....

Tech Mahindra Data Engineer Interview Questions

+2 More Questions with Expert Answers

Other Companies

Tech Mahindra Data Engineer Interview Questions

+2 More Questions with Expert Answers

Other Companies