Interview Questions

Schema evolution - techniques for handling schema changes in PySpark

Setting Dependencies for Tasks in DAG

Share your experience in working with big data technologies such as Hadoop, Spark, or AWS EMR. How have you leveraged these tools in your previous projects?

Solve 7-8 data processing questions using PySpark on F1 Racing Data

Solve the dataset transformation using PySpark.

Solve the grade assignment problem using a UDF in PySpark.

Spark Architecture - Components include Driver, Executors, Cluster Manager, and Tasks

Spark Configurations for Large-Scale Jobs

Spark Execution Flow - describe

Spark Executor Management: 10 workers, 100GB RAM, 25 cores - number of executors, size, OOM in Driver

Spark Optimization - broadcast joins, caching, coalescing, predicate pushdown, AQE

Spark Optimizations: skewed joins, broadcast joins, Catalyst Optimizer, repartition vs coalesce

Spark Session Command - how to create

Spark Streaming - streaming data handling and file mounting techniques

Spark Submit - command syntax

Spark Tungsten & Catalyst Optimizer

Steps to link a Databricks notebook to an ADF pipeline

Trade-offs between batch processing (Spark) vs. real-time streams (Kafka)

Usage of UDFs?

Walk through how you would debug the data ingestion process to identify slow stages.