Interview Questions

PySpark Coding Challenge - dataset with 4-5 columns, solve data processing problem on paper

PySpark Coding Challenge: Transform input dataset with columns id, dob, name to add age, firstname, lastname

Read CSV, filter, and write to table using PySpark

Running Tasks in Parallel

Salting Implementation - provide example

Schema evolution - techniques for handling schema changes in PySpark

Setting Dependencies for Tasks in DAG

Share your experience in working with big data technologies such as Hadoop, Spark, or AWS EMR. How have you leveraged these tools in your previous projects?

Solve 7-8 data processing questions using PySpark on F1 Racing Data

Solve the dataset transformation using PySpark.

Solve the grade assignment problem using a UDF in PySpark.

Spark Architecture - Components include Driver, Executors, Cluster Manager, and Tasks

Spark Configurations for Large-Scale Jobs

Spark Execution Flow - describe

Spark Executor Management: 10 workers, 100GB RAM, 25 cores - number of executors, size, OOM in Driver

Spark Optimization - broadcast joins, caching, coalescing, predicate pushdown, AQE

Spark Optimizations: skewed joins, broadcast joins, Catalyst Optimizer, repartition vs coalesce

Spark Session Command - how to create

Spark Streaming - streaming data handling and file mounting techniques

Spark Submit - command syntax