Real questions from top companies in Spark/Big Data
What is Shuffle and How to Handle It in Spark
What role would Kafka or similar event-driven platforms play in your architecture?
Write PySpark code to extract data from a CSV and create a table.
Write PySpark code to filter and count records.
Write PySpark code to filter records based on specific conditions and add a calculated column.
Write PySpark code to save a DataFrame in Parquet format to an S3 bucket.
Write a PySpark code snippet to filter rows with a specific condition.
Write a PySpark job that calculates the number of unique users who logged in per day, but exclude any logins from inactive users listed in a separate file.
Write a PySpark script to check for missing values and duplicate rows in a DataFrame. How would you ensure data quality before saving it to a storage system?
Write a PySpark script to filter out invalid records from a dataset and calculate the average for a specific column, ensuring the schema is strictly defined at runtime.
Write a PySpark script to process data stored in Delta format and transform it into Parquet.
Write a PySpark script to read a CSV file, filter rows where the age column is less than 18, and write the result to a new CSV file.
Write a Spark job to count word occurrences from an S3 dataset.
Write a complete PySpark program from import statements to the stop statement, covering transformations and actions.
Write a transformation in PySpark to join and clean multiple raw input sources
Write code to read data from Delta Lake in S3 and perform upsert based on primary key
Write maintainable, efficient Pandas or PySpark code.
Writing Excel sheets to Delta tables in Databricks
You are given 10 worker machines with 100 GB RAM and 25 CPU cores. How would you determine the number of executors and the size of each executor?
Your Kafka producer schema has changed, and the new data includes additional fields. How would you ensure backward compatibility using Schema Registry while consuming data from the same topic?
Type or paste your answer to any of these questions and our AI Coach scores it, highlights gaps, and rewrites it at FAANG quality. Free to try.