Question 1

What are Airflow Operators? Give examples.

Accepted Answer

Airflow Operators define a single unit of work in a DAG—each operator performs one atomic, idempotent task. **Why they matter**: They encapsulate work so DAGs remain declarative and schedulable; the scheduler doesn't need to understand task logic. **Examples**: BashOperator, PythonOperator, SqlOperator, HTTPOperator, DockerOperator, KubernetesPodOperator, Sensor. **Scalability**: Heavy logic should live in external scripts or services; operators should only orchestrate....

Question 2

How do you remove duplicate rows in BigQuery?

Accepted Answer

Approach: Use ROW_NUMBER() OVER (PARTITION BY dedup_keys ORDER BY tie_breaker) to define which row to keep; filter rn = 1. Preferred pattern: CREATE OR REPLACE TABLE ... AS SELECT * EXCEPT(rn) FROM (SELECT *, ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1. Why CREATE OR REPLACE over DELETE: BigQuery is columnar; DELETE is a rewrite under the hood. For large tables, CREATE OR REPLACE is a single scan+write vs DELETE's read-modify-write....

Question 3

What are the key components of AWS Glue, and how do they work together?

Accepted Answer

Glue Catalog: Central metadata store (Hive-compatible)—enables querying S3 data via Athena/Redshift Spectrum without moving it. Glue Crawlers: Schema discovery and Catalog population—useful for ad-hoc sources; at scale, prefer schema-as-code to avoid crawler cost and drift. Glue ETL Jobs: Serverless Spark for transforms; auto-scaling, pay-per-DPU. Glue DataBrew: Visual prep for non-engineers. Glue Schema Registry: Schema evolution for streaming (Kafka, Kinesis)....

Question 4

What is Azure Data Factory (ADF), and what are its main components?

Accepted Answer

ADF is a cloud-native data integration service for orchestration and movement. Components: Pipelines (logical groups of activities), Activities (Copy, Lookup, Databricks, Data Flow), Datasets (structure definitions), Linked Services (connection configs), Triggers (schedule or event-based), Integration Runtime (IR—compute for execution). Flow: Linked Service -> Dataset -> Activity -> Pipeline -> Trigger....

Question 5

What is Snowflake's architecture, and why is it unique?

Accepted Answer

**Section 1 — The Context (The 'Why')**
Traditional data warehouses collapse under elastic concurrency: fixed clusters either over-provision (cost) or under-provision (queuing). Storage-compute coupling means scaling queries requires scaling storage nodes....

Question 6

What is the difference between S3 and HDFS?

Accepted Answer

S3: Object storage—durable, highly available, decoupled from compute. Pay-per-use, virtually unlimited scale. Eventual consistency (now strong for new overwrites). No data locality. HDFS: Distributed file system—block-based, colocated with compute nodes. Data locality reduces network I/O; strong consistency. Requires cluster management. Why it matters: S3 enables cloud-native, serverless patterns (Lambda, Glue, Athena); HDFS optimizes for batch processing where locality matters....

Question 7

What is the role of AWS Lambda in a data engineering pipeline?

Accepted Answer

Lambda is serverless compute for event-driven, short-lived tasks. Roles: (1) Orchestration: Trigger Glue, EMR, or Step Functions when files land (S3), events arrive (Kinesis), or schedules fire. (2) Light transforms: JSON→Parquet, small aggregations, API calls. (3) Alerts: Validate data, send Slack/PagerDuty on anomalies. Why: No server management; scale to thousands of concurrent invocations; pay per request and duration. Limitations: 15-min timeout, 10GB memory, 6MB payload (sync)....

Question 8

What is the role of the Integration Runtime (IR) in ADF?

Accepted Answer

IR is the compute layer that executes ADF activities—data movement, transformations, and external calls. Types: (1) Azure IR: Managed, cloud-native—for cloud-to-cloud copies, Data Flows, Databricks calls. No maintenance. (2) Self-hosted IR: Install on your VM or container—for on-prem, VNet, or private data sources. You manage scaling and uptime. (3) Azure-SSIS IR: For running SSIS packages in Azure. Why it matters: Data never flows through ADF; it flows through IR. Network topology (cloud vs....

Tech Mahindra Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies

Tech Mahindra Data Engineer Interview Questions

Reading isn't practice. Get AI feedback on your answers.

Other Companies