Spark & Big Data questions from Infosys data engineering interviews.
These spark & big data questions are sourced from Infosys data engineering interviews. Each includes an expert-level answer. This set leans toward the medium-difficulty band most real interviews actually live in (11 of 19). Recurring themes are spark, partition, and sql — these patterns appear most often in real interviews and reward the deepest preparation. Many of these questions also surface at Altimetrik and American Express, so the preparation transfers across companies. Average answer is around 1 minute of reading — plan roughly 1 hour to work through the full set thoughtfully.
This collection contains 19 curated questions: 3 easy, 11 medium, and 5 hard. The distribution skews toward harder problems, reflecting the depth expected in senior-level interviews.
The most frequently tested areas in this set are spark (17), partition (10), sql (9), python (4), join (4), and optimization (3). Focusing on these topics will give you the highest return on your preparation time.
Start with the easy questions to warm up and solidify fundamentals. Medium-difficulty questions form the bulk of real interviews — spend the most time here and practice explaining your reasoning out loud. Hard questions often appear in senior and staff-level rounds; attempt them after you're comfortable with the basics. For each question, try answering before revealing the solution. Use our AI Mock Interview to simulate real interview conditions and get instant feedback on your responses.
What is the difference between SparkSession and SparkContext in Spark?
Architecturally, how would you justify or challenge Hadoop vs. a cloud-native data lake (S3 + EMR/Databricks) for a greenfield enterprise data platform? Discuss scalability ceilings, cost model trade-offs, and operational complexity.
How would you read data from a web API using PySpark?
What is broadcasting in Spark, and why is it used? Can you give an example of its use?
What is the difference between map and flatMap in Spark, and when would you use each?
What is the purpose of the Bronze, Silver, and Gold layers in a data pipeline?
What work is done by the executor memory in Spark?
When and how do you use Broadcast Join?
Why is SparkSession used in Spark 2.0 and later versions?
Write a Python script to find the count of each word in a text file using Spark.
Write the PySpark code to find the second highest salary in each department.
Can you explain the concept of incremental loading in Sqoop and how to use it for job processing?
Can you explain the concept of mappers in Spark, and how are they used in data transformations?
How would you move a file to another path in Databricks File System (DBFS)?
How would you read data from an RDBMS using Spark? Provide the syntax.
What Hadoop command would you use to merge multiple files into one?
What is YARN, and how does it manage resources in a Hadoop ecosystem?
What is the difference between managed and external tables in Hive or Spark SQL?
What performance tuning techniques do you apply in both Sqoop and Spark to optimize their execution?
Get full access to 1,800+ expert answers, AI mock interviews, and personalized progress tracking.