Question 1

What is the difference between repartition and coalesce in Apache Spark?

Accepted Answer

**Repartition(n)**: Performs a full shuffle to redistribute data across exactly `n` partitions. Can increase or decrease partition count. Uses hash partitioning by default—all rows are exchanged across the network.

**Coalesce(n)**: Merges existing partitions into fewer partitions without a full shuffle. Only decreases partition count....

Question 2

What is the difference between SparkSession and SparkContext in Spark?

Accepted Answer

**SparkContext** (Spark 1.x): Low-level entry point for RDD operations. Manages cluster connections, configuration, and RDD creation. One active SparkContext per JVM. RDD-only.

**SparkSession** (Spark 2.0+): Unified entry point subsuming SparkContext, SQLContext, HiveContext, StreamingContext. Provides DataFrame, Dataset, SQL, and Structured Streaming APIs....

Question 3

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Accepted Answer

**Narrow transformations**: Each input partition maps to at most one output partition. No shuffle. Examples: map, filter, flatMap, mapPartitions.

**Wide transformations**: Require data from multiple input partitions to produce one output partition. Trigger shuffle. Examples: groupByKey, reduceByKey, join, distinct, repartition.

**Architectural Logic (Why This Matters)**: Spark pipelines narrow transformations and executes them in a single stage....

Question 4

What are Airflow Operators? Give examples.

Accepted Answer

Airflow Operators define a single unit of work in a DAG—each operator performs one atomic, idempotent task. **Why they matter**: They encapsulate work so DAGs remain declarative and schedulable; the scheduler doesn't need to understand task logic. **Examples**: BashOperator, PythonOperator, SqlOperator, HTTPOperator, DockerOperator, KubernetesPodOperator, Sensor. **Scalability**: Heavy logic should live in external scripts or services; operators should only orchestrate....

Question 5

Write a Python function to check if a string is a palindrome.

Accepted Answer

**Architectural logic**: A palindrome reads the same forwards and backwards. We need to normalize (case, non-alphanumeric) and compare. **Approach 1 (string ops)**: `cleaned = "".join(c.lower() for c in s if c.isalnum()); return cleaned == cleaned[::-1]`—O(n) time, O(n) space. **Approach 2 (two-pointer)**: Compare from both ends; O(n) time, O(1) space if not normalizing....

Question 6

What is the difference between a list and a tuple in Python?

Accepted Answer

List: Mutable, []; tuple: immutable, (). Why it matters: Mutability drives use—lists for collections that change; tuples for fixed data, dict keys (hashable), multiple return values. Performance: Tuples are slightly faster (less overhead, fixed size). Hashability: Tuples can be dict keys/set members; lists cannot. In data pipelines: Tuples for schema-like rows (column names); lists for buffers, accumulators....

Question 7

Explain the difference between shallow copy and deep copy in Python.

Accepted Answer

Shallow (copy.copy()): New top-level object; nested objects are references. Nested mutations affect original. Deep (copy.deepcopy()): Recursive copy; fully independent. Why it matters: Shallow is O(n) for top level only; deep is O(n) for entire structure—can be slow for large nested dicts. Use shallow when: No nested mutables or shared refs OK. Use deep when: Need full isolation (e.g., config that will be modified)....

Question 8

Write a Python function to find the first non-repeating character in a string.

Accepted Answer

Approach: Two-pass—count chars, then find first with count 1. Code: def first_non_repeating(s): counts = {}; [counts.update({c: counts.get(c, 0) + 1}) for c in s]; return next((c for c in s if counts[c] == 1), None). Or: from collections import Counter; counts = Counter(s); return next((c for c in s if counts[c] == 1), None). Complexity: O(n) time, O(k) space. Why: Single pass can't know if char is unique until full scan....

Question 9

What are decorators in Python, and how do they work?

Accepted Answer

Decorator: Higher-order function that wraps another function; @decorator syntax. Implementation: def timer(f): def wrapper(*args, **kwargs): start=time(); r=f(*args,**kwargs); print(time()-start); return r; return wrapper. Why: Cross-cutting concerns—logging, retries, auth, caching—without cluttering business logic. In data pipelines: @retry, @logged, @rate_limited on API fetchers. Scalability: Decorators add indirection; overuse can make stack traces deep....

Question 10

Explain the difference between args and kwargs in Python.

Accepted Answer

*args: Variable positional args, received as tuple. **kwargs: Variable keyword args, received as dict. Why: Flexible signatures—wrappers, decorators, config-driven functions. Example: def run_pipeline(*sources, **config): ... allows run_pipeline('s3://a','s3://b', parallelism=4). Order: (regular, *args, keyword-only, **kwargs). Pitfall: Passing through—func(*args, **kwargs)—preserves interface. In data pipelines: **kwargs for optional connector params (region, timeout, retries)....

Python Interview Questions

Difficulty Breakdown

Key Topics Covered

How to Use This Guide

Companies asking these questions

All 60 Questions

More Interview Prep Guides

Unlock All Expert Answers