Real questions from top companies in Spark/Big Data · medium
What work is done by the executor memory in Spark?
When and how do you use Broadcast Join?
Write a Python script to find the count of each word in a text file using Spark.
Write the PySpark code to find the second highest salary in each department.
Broadcast Joins and Shuffle Merge Joins?
Can you explain the concept of mappers in Spark, and how are they used in data transformations?
Code a simple PySpark job to read a JSON file, filter records, and write output in Parquet format.
How do you handle out-of-memory errors in Spark jobs?
How do you handle very large datasets in Spark to ensure scalability and efficiency?
Provide specific examples of challenges faced with PySpark and SQL and solutions implemented.
Split a DataFrame such that even numbers appear in one column and odd numbers in another
Steps to mount storage in Databricks.
Transformation vs. Action in PySpark?
What Hadoop command would you use to merge multiple files into one?
What are the limitations of the REORG command with respect to large datasets?
What causes Out of Memory (OOM) issues in Databricks, and how do you resolve them?
What is Shuffle and How to Handle It in Spark
Write PySpark code to extract data from a CSV and create a table.
Write PySpark code to save a DataFrame in Parquet format to an S3 bucket.
Write a PySpark job that calculates the number of unique users who logged in per day, but exclude any logins from inactive users listed in a separate file.
Type or paste your answer to any of these questions and our AI Coach scores it, highlights gaps, and rewrites it at FAANG quality. Free to try.