Question 1

Core services of AWS used in data engineering?

Accepted Answer

Why this stack: Each service has a role—ingestion, storage, compute, serving. Architectural logic: S3 (lake); EMR/Glue (Spark); Lambda (event-driven); Kinesis (streaming); DMS (CDC); Redshift (warehouse); Athena (serverless SQL); MWAA (Airflow); Glue Catalog (metadata); IAM/KMS (security); CloudWatch (monitoring). Trade-offs: Glue vs. EMR—Glue serverless, EMR more control. Athena vs. Redshift—Athena pay per query; Redshift for interactive....

Question 2

Describe how to set up retries and timeout for tasks in Cloud Composer.

Accepted Answer

Architectural logic: Retries handle transient failures; timeouts prevent hung tasks from blocking DAGs. Configuration: default_args or per-task: retries, retry_delay, retry_exponential_backoff, execution_timeout. Example: retries=3, retry_delay=timedelta(minutes=2), retry_exponential_backoff=True, execution_timeout=timedelta(minutes=45). Why exponential backoff: Reduces thundering herd on downstream services. Scalability: Too many retries on large DAGs can delay subsequent runs....

Question 3

Describe the use of side inputs in Dataflow.

Accepted Answer

Architectural logic: Side inputs = small lookup broadcast to all workers; avoid expensive shuffle. Use case: Enrich events with product catalog or user segments. Main PCollection + side input (AsDict/AsIter) → Map with lookup. Why: No large join; each worker has full lookup. Scalability: Side input must fit in memory; large lookups use CoGroupByKey. Cost: Broadcast is cheap; oversized side input causes OOM....

Question 4

Explain the key components of Apache Beam in the context of Google Dataflow.

Accepted Answer

Architectural components: PCollection (bounded/unbounded dataset), PTransform (Map, Filter, GroupByKey), Pipeline (graph), Runner (Dataflow). Side inputs for lookups; watermarks for event-time. Flow: ReadFromPubSub → Map → WindowInto → GroupByKey → WriteToBigQuery. Why Beam: Portable; batch and stream....

Question 5

Explain the role of Airflow DAGs in Cloud Composer.

Accepted Answer

Architectural role: DAGs = workflow definition (tasks + dependencies). Composer syncs from GCS bucket. DAGs orchestrate ETL—schedule, retry, monitor. Example: daily_etl, schedule 0 6 * * *. Best practice: DAGs in Git; variables/connections for config; retries/timeouts; KubernetesExecutor for isolation.

Question 6

How do you optimize resource allocation in a Dataflow job to reduce costs?

Accepted Answer

Strategies: (1) Right-size workers (machineType, numWorkers). (2) Batch vs streaming—batch cheaper. (3) Reduce shuffle; optimize keys. (4) Autoscaling with max cap. (5) Committed use. Trade-off: Under-provision = slow; over = costly....

Question 7

How would you secure sensitive credentials in Cloud Composer workflows?

Accepted Answer

**Why secrets management**: Credentials in DAG code get committed to Git—instant breach. Airflow Variables and Connections stored in metadata DB are better but still risk exposure. **Architecture**: Google Secret Manager as source of truth. Configure Airflow Secrets Backend (CloudSecretManagerBackend) so Variable.get() and Connection lookups resolve from Secret Manager. Create secrets in Secret Manager; reference by path....

Aarete Cloud & Tools Interview Questions

Companies asking these questions

All 7 Questions

More Interview Prep Guides

Get All Answers in PDF Format

Aarete Cloud & Tools Interview Questions

Companies asking these questions

All 7 Questions

More Interview Prep Guides

Get All Answers in PDF Format