Question 1

What is the difference between groupByKey and reduceByKey in Spark?

Accepted Answer

**groupByKey()**: Shuffles all (key, value) pairs to group values per key. Transfers O(total_values) over the network. No local aggregation—you combine values afterward. High memory and network cost.

**reduceByKey(func)**: Performs local reduce (e.g., sum) on each partition before shuffle. Shuffles only O(unique_keys) aggregated values. Combines locally first, then across partitions.

**Architectural Logic (Why reduceByKey Wins)**: Shuffle is the bottleneck....

Question 2

Demonstrate the difference between DENSE_RANK() and RANK()

Accepted Answer

**RANK()**: Same rank for ties; skips subsequent ranks (e.g., 1, 2, 2, 4, 5). **DENSE_RANK()**: Same rank for ties; no gaps (e.g., 1, 2, 2, 3, 4). **Why it matters**: RANK preserves "position" semantics (e.g., 4th place); DENSE_RANK gives consecutive integers useful for filtering (e.g., TOP 10). **Example**: `SELECT name, salary, RANK() OVER (ORDER BY salary DESC) AS rk, DENSE_RANK() OVER (ORDER BY salary DESC) AS dense_rk FROM employee`....

Question 3

Write a Python function to check if a string is a palindrome.

Accepted Answer

**Architectural logic**: A palindrome reads the same forwards and backwards. We need to normalize (case, non-alphanumeric) and compare. **Approach 1 (string ops)**: `cleaned = "".join(c.lower() for c in s if c.isalnum()); return cleaned == cleaned[::-1]`—O(n) time, O(n) space. **Approach 2 (two-pointer)**: Compare from both ends; O(n) time, O(1) space if not normalizing....

Question 4

Implement a Spark job to find the top 10 most frequent words in a large text file.

Accepted Answer

Core logic: read text → split → explode → filter empty → groupBy → count → orderBy desc → limit 10. Code: from pyspark.sql import functions as F; df = spark.read.text("path/to/file.txt"); words = df.select(F.explode(F.split(F.col("value"), "\s+")).alias("word")); top10 = words.filter(F.length(F.col("word")) > 0).groupBy("word").count().orderBy(F.desc("count")).limit(10). **Why \s+**: Handles multiple spaces/tabs; more robust than single space....

Question 5

Describe a real-world use case for using Step Functions with Lambda in a data workflow.

Accepted Answer

Use case: ML inference and reporting pipeline. Raw events land in S3; Lambda validates; Step Functions orchestrates: preprocessing Lambda → external ML API (wait) → result Lambda writes to DynamoDB/S3 → Slack summary. Why Step Functions + Lambda: Lambda = stateless, short compute; Step Functions = state, retries, branching, observability. Architectural trade-off: Express Workflows for high-volume, short runs (cheaper); Standard for long-running, complex branching....

Question 6

Describe using Step Functions to handle retries and error notifications.

Accepted Answer

Architectural logic: Retry handles transient failures; Catch routes terminal failures to notifications or DLQ. Config: Retry with ErrorEquals, IntervalSeconds, BackoffRate, MaxAttempts; Catch with Next (e.g., NotifyFailure). Example: Lambda.ServiceException retry 3x with exponential backoff; States.ALL → NotifyFailure (SNS). Why: Observability; no lost failures....

Question 7

Explain how Access Control Lists (ACLs) can affect IAM role permissions.

Accepted Answer

Architectural logic: S3 has IAM and ACLs (legacy). Both must allow—ACLs can deny even if IAM allows. ACLs = per-object/bucket grants. IAM = identity-based. Trade-off: ACLs add complexity; prefer bucket policies + IAM....

Question 8

Explain how Step Functions integrate with other AWS services.

Accepted Answer

Architectural logic: Step Functions has native integrations—Lambda, Glue, ECS, SNS, SQS, DynamoDB, SageMaker, EventBridge. Each = Task state with service ARN and payload. Flow: Lambda validate → Glue ETL → Lambda write DynamoDB. Why: No polling; optimized integrations....

Capco Data Engineer Interview Questions

+17 More Questions with Expert Answers

Other Companies

Capco Data Engineer Interview Questions

+17 More Questions with Expert Answers

Other Companies