#spark

What is the difference between repartition and coalesce in Apache Spark?

What is the difference between SparkSession and SparkContext in Spark?

What are traits in Scala, and how are they different from classes?

What is the difference between cache() and persist() in Spark? When would you use each?

What is the difference between groupByKey and reduceByKey in Spark?

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

Explain the differences between Data Warehouse, Data Lake, and Delta Lake

What is the difference between partitioning and bucketing in Spark, and when would you use bucketing?

What strategies can you use to handle skewed data in Spark?

Briefly introduce yourself and walk us through your journey as a Data Engineer so far.

What is the difference between a primary key and a unique key?

Can you explain the architecture of Apache Spark and its components?

Describe the difference between Spark RDDs, DataFrames, and Datasets.

Explain the difference between Spark's map() and flatMap() transformations.

How does Spark's Catalyst Optimizer work? Explain its stages.

How do you handle late-arriving data in Spark Structured Streaming?

What is the difference between Managed and External tables in Hive/Spark?

What is the small-file problem in Spark, and how do you solve it?

Explain the concept of Broadcast Join in Spark. When should it be used?

How do you optimize Spark jobs for better performance? Mention at least 5 techniques.