Question 1

What is the difference between SparkSession and SparkContext in Spark?

Accepted Answer

**SparkContext** (Spark 1.x): Low-level entry point for RDD operations. Manages cluster connections, configuration, and RDD creation. One active SparkContext per JVM. RDD-only.

**SparkSession** (Spark 2.0+): Unified entry point subsuming SparkContext, SQLContext, HiveContext, StreamingContext. Provides DataFrame, Dataset, SQL, and Structured Streaming APIs....

Question 2

What is the difference between partitioning and bucketing in Spark, and when would you use bucketing?

Accepted Answer

**Partitioning**: Physically divides data by column values (e.g., date, region); enables partition pruning; one directory per partition value. **Bucketing**: Divides data within a partition into a fixed number of files via hash of bucketing column(s); co-locates same-key rows. **When to bucket**: Frequent joins or group-bys on a column (e.g., user_id). Same bucket count on both sides enables sort-merge join without shuffle....

Question 3

Write a Python function to check if a string is a palindrome.

Accepted Answer

**Architectural logic**: A palindrome reads the same forwards and backwards. We need to normalize (case, non-alphanumeric) and compare. **Approach 1 (string ops)**: `cleaned = "".join(c.lower() for c in s if c.isalnum()); return cleaned == cleaned[::-1]`—O(n) time, O(n) space. **Approach 2 (two-pointer)**: Compare from both ends; O(n) time, O(1) space if not normalizing....

Question 4

When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.

Accepted Answer

DataFrame is an untyped collection of Row objects with schema at runtime; Dataset[T] is typed with compile-time safety. In Scala, DataFrame = Dataset[Row]. Architectural why: Dataset enables domain modeling (e.g., Dataset[Order])—catch errors at compile time, better IDE support, and Catalyst can optimize typed encoders. Scalability: both use Tungsten and Catalyst; Dataset adds encoder overhead but marginal for most workloads. Portability trade-off: PySpark has only DataFrame—no typed Dataset....

Question 5

Design a cost-aware resource strategy for a Databricks workload with spiky and batch jobs. Explain Dynamic Resource Allocation, when to disable it, and how min/max executors and spot instances affect cost and SLAs.

Accepted Answer

**Section 1 — The Context (The 'Why')**
Databricks workload cost explodes when clusters run idle, jobs are over-provisioned, or spot preemption causes thrashing. The challenge is aligning DPU allocation to actual parallelism while maintaining SLA....

Question 6

Command to Read JSON Data and Options

Accepted Answer

**Spark**: spark.read.json('path') or spark.read.format('json').load('path'). **Options**: schema (explicit StructType for perf), multiLine (pretty-printed), dateFormat, primitivesAsString. **Nested**: Provide schema or from_json. **Streaming**: readStream.schema(schema).json(path)....

Question 7

Daily Data Volume - quantify

Accepted Answer

**Quantify**: Example—500GB raw, 200GB curated daily. Peak 2TB month-end. Include records and bytes. **Source**: Pipeline metrics (Glue, Dataflow). **Why**: Sizes infrastructure; capacity planning; cost....

Question 8

Describe a project you worked on, focusing on the data pipeline and your role.

Accepted Answer

**STAR**: **Situation**: Needed clickstream analytics. **Task**: Build pipeline. **Action**: Designed Kafka ingest, Flink processing, ClickHouse storage. Led design; implemented core components. **Result**: Sub-minute latency; 10M events/day....

Question 9

What is Multiline option in JSON?

Accepted Answer

multiline=true: One JSON object can span multiple lines. Default (false): one object per line (NDJSON). Spark: .option('multiline','true').json(path). WHY: Pretty-printed API responses. Caution: Slower; mixed single/multi can fail....

Question 10

Case Class and StructType Syntax

Accepted Answer

**Case Class (Scala):** `case class Person(name: String, age: Int)`—immutable, auto equals/hashCode. **StructType (Spark):** `StructType([StructField('name', StringType()), ...])`. **Mapping:** Case classes → Spark rows. **When:** StructType for dynamic/Python; Case Class for type-safe Scala....

LTIMindtree Data Engineer Interview Questions

Companies asking these questions

All 29 Questions

More Interview Prep Guides

Get All Answers in PDF Format

LTIMindtree Data Engineer Interview Questions

Companies asking these questions

All 29 Questions

More Interview Prep Guides

Get All Answers in PDF Format