Question 1

What is the difference between SparkSession and SparkContext in Spark?

Accepted Answer

**SparkContext** (Spark 1.x): Low-level entry point for RDD operations. Manages cluster connections, configuration, and RDD creation. One active SparkContext per JVM. RDD-only.

**SparkSession** (Spark 2.0+): Unified entry point subsuming SparkContext, SQLContext, HiveContext, StreamingContext. Provides DataFrame, Dataset, SQL, and Structured Streaming APIs....

Question 2

What are traits in Scala, and how are they different from classes?

Accepted Answer

**Traits**: Interface-like constructs that can define abstract and concrete methods/fields. Support multiple inheritance of type. Mixed in via `with`.

**Classes**: Define objects with state and behavior. Single inheritance; one superclass.

**Key Differences**: Traits enable composition; classes define core logic. Traits can be partially implemented; classes hold primary behavior....

Question 3

How do you handle data security and compliance in a cloud environment?

Accepted Answer

Security is layered: (1) Encryption: At rest (KMS-managed keys, SSE-S3, Azure Storage encryption) and in transit (TLS). Why: Compliance (GDPR, HIPAA) and breach mitigation. Trade-off: Key management adds latency and complexity; managed services reduce operational burden. (2) Access: Least-privilege IAM, role-based access, no long-lived keys in code. Use VPC/VNet for network isolation; private endpoints for data stores....

Question 4

How would you read data from a web API? What steps would you follow after reading the data?

Accepted Answer

**Why this matters**: APIs are external dependencies—unreliable, rate-limited, schema-evolving. Production ingestion must be resilient, auditable, and idempotent. **Steps**: (1) **Contract & auth**: Define schema (Pydantic, JSON Schema); store credentials in secrets manager (Vault, AWS Secrets). OAuth tokens need refresh logic. (2) **Pagination & rate limiting**: Offset/cursor-based; loop until empty. Respect Retry-After; implement exponential backoff....

Question 5

Architecturally, how would you justify or challenge Hadoop vs. a cloud-native data lake (S3 + EMR/Databricks) for a greenfield enterprise data platform? Discuss scalability ceilings, cost model trade-offs, and operational complexity.

Accepted Answer

Hadoop's architecture centers on HDFS (storage) + YARN (resource scheduling) + MapReduce (compute model). Components: Namenode (single point for metadata—scalability ceiling for namespace); Datanodes (block storage, 128MB default blocks, 3x replication); ResourceManager/NodeManagers for CPU/memory allocation. Why it matters architecturally: HDFS scales linearly with nodes but metadata scale is bounded by Namenode heap; YARN allows multi-tenant workloads but adds operational overhead....

Question 6

How would you read data from a web API using PySpark?

Accepted Answer

PySpark has no native API source; the pattern is driver fetch or executor fetch. **Approaches**: (1) **Driver + parallelize**: data = requests.get(url).json(); df = spark.createDataFrame(data). Scales to API response size (typically MBs); driver is bottleneck. (2) **mapPartitions on executor**: Pass partition of IDs to each task; each calls API. Scales to many IDs but risks rate limiting and API abuse. (3) **Orchestrator + landing**: Airflow/Prefect fetches API → lands to S3/GCS → Spark reads....

Question 7

What is broadcasting in Spark, and why is it used? Can you give an example of its use?

Accepted Answer

Broadcasting sends a small table replica to every executor so joins execute locally without shuffle. **Why it matters**: A shuffle join moves both sides across the network; broadcast join moves only the small side once from driver to executors. **Scalability trade-off**: The small table must fit in executor memory; exceeding this causes OOM. As cluster size grows, more copies exist (N executors × table size), so large broadcasts waste memory....

Question 8

What is the difference between map and flatMap in Spark, and when would you use each?

Accepted Answer

**map**: 1 input element → 1 output element; output collection has same cardinality. **flatMap**: 1 input element → 0 or more output elements; results are flattened into a single collection. **Why it matters**: flatMap controls cardinality explosion (e.g., tokenizing a line into N words) and avoids nested structures that would require a separate explode step....

Infosys Data Engineer Interview Questions

+15 More Questions with Expert Answers

Other Companies

Infosys Data Engineer Interview Questions

+15 More Questions with Expert Answers

Other Companies