Question 1

What are Airflow Operators? Give examples.

Accepted Answer

Airflow Operators define a single unit of work in a DAG—each operator performs one atomic, idempotent task. **Why they matter**: They encapsulate work so DAGs remain declarative and schedulable; the scheduler doesn't need to understand task logic. **Examples**: BashOperator, PythonOperator, SqlOperator, HTTPOperator, DockerOperator, KubernetesPodOperator, Sensor. **Scalability**: Heavy logic should live in external scripts or services; operators should only orchestrate....

Question 2

How do you remove duplicate rows in BigQuery?

Accepted Answer

Approach: Use ROW_NUMBER() OVER (PARTITION BY dedup_keys ORDER BY tie_breaker) to define which row to keep; filter rn = 1. Preferred pattern: CREATE OR REPLACE TABLE ... AS SELECT * EXCEPT(rn) FROM (SELECT *, ROW_NUMBER() OVER (...) AS rn ...) WHERE rn = 1. Why CREATE OR REPLACE over DELETE: BigQuery is columnar; DELETE is a rewrite under the hood. For large tables, CREATE OR REPLACE is a single scan+write vs DELETE's read-modify-write....

Question 3

What are the key components of AWS Glue, and how do they work together?

Accepted Answer

Glue Catalog: Central metadata store (Hive-compatible)—enables querying S3 data via Athena/Redshift Spectrum without moving it. Glue Crawlers: Schema discovery and Catalog population—useful for ad-hoc sources; at scale, prefer schema-as-code to avoid crawler cost and drift. Glue ETL Jobs: Serverless Spark for transforms; auto-scaling, pay-per-DPU. Glue DataBrew: Visual prep for non-engineers. Glue Schema Registry: Schema evolution for streaming (Kafka, Kinesis)....

Question 4

What is Azure Data Factory (ADF), and what are its main components?

Accepted Answer

ADF is a cloud-native data integration service for orchestration and movement. Components: Pipelines (logical groups of activities), Activities (Copy, Lookup, Databricks, Data Flow), Datasets (structure definitions), Linked Services (connection configs), Triggers (schedule or event-based), Integration Runtime (IR—compute for execution). Flow: Linked Service -> Dataset -> Activity -> Pipeline -> Trigger....

Question 5

What is Snowflake's architecture, and why is it unique?

Accepted Answer

**Section 1 — The Context (The 'Why')**
Traditional data warehouses collapse under elastic concurrency: fixed clusters either over-provision (cost) or under-provision (queuing). Storage-compute coupling means scaling queries requires scaling storage nodes....

Question 6

What is the difference between S3 and HDFS?

Accepted Answer

S3: Object storage—durable, highly available, decoupled from compute. Pay-per-use, virtually unlimited scale. Eventual consistency (now strong for new overwrites). No data locality. HDFS: Distributed file system—block-based, colocated with compute nodes. Data locality reduces network I/O; strong consistency. Requires cluster management. Why it matters: S3 enables cloud-native, serverless patterns (Lambda, Glue, Athena); HDFS optimizes for batch processing where locality matters....

Question 7

What is the role of AWS Lambda in a data engineering pipeline?

Accepted Answer

Lambda is serverless compute for event-driven, short-lived tasks. Roles: (1) Orchestration: Trigger Glue, EMR, or Step Functions when files land (S3), events arrive (Kinesis), or schedules fire. (2) Light transforms: JSON→Parquet, small aggregations, API calls. (3) Alerts: Validate data, send Slack/PagerDuty on anomalies. Why: No server management; scale to thousands of concurrent invocations; pay per request and duration. Limitations: 15-min timeout, 10GB memory, 6MB payload (sync)....

Question 8

What is the role of the Integration Runtime (IR) in ADF?

Accepted Answer

IR is the compute layer that executes ADF activities—data movement, transformations, and external calls. Types: (1) Azure IR: Managed, cloud-native—for cloud-to-cloud copies, Data Flows, Databricks calls. No maintenance. (2) Self-hosted IR: Install on your VM or container—for on-prem, VNet, or private data sources. You manage scaling and uptime. (3) Azure-SSIS IR: For running SSIS packages in Azure. Why it matters: Data never flows through ADF; it flows through IR. Network topology (cloud vs....

Question 9

Detail examples of inner, outer, left, and right joins.

Accepted Answer

**Architectural Logic**: Join choice directly impacts query cost, data correctness, and downstream semantics. INNER keeps only matching rows—ideal when referential integrity is enforced and you want to exclude orphans; minimizes shuffle and output size. LEFT preserves all from A with optional B—use when B is a "dimension" that may not exist (e.g., optional user attributes); RIGHT is mirror of LEFT; FULL OUTER preserves both sides—expensive due to bilateral NULL expansion....

Question 10

Difference Between Internal and External Tables in BigQuery

Accepted Answer

**Architectural Logic**: Internal (native) tables: BigQuery owns metadata and storage (Colossus). Columnar layout, partitioning, clustering, automatic optimization. Dropping = data deleted. External tables: BigQuery owns metadata only; data lives in GCS, Drive, Bigtable. Dropping = metadata deleted, data persists. **Why**: Internal enables DML, streaming inserts, partitioning strategies, cost-effective storage classes....

Tech Mahindra Data Engineer Interview Questions

Companies asking these questions

All 22 Questions

More Interview Prep Guides

Get All Answers in PDF Format

Tech Mahindra Data Engineer Interview Questions

Companies asking these questions

All 22 Questions

More Interview Prep Guides

Get All Answers in PDF Format