#partition

Tell me about yourself and your experience.

What is the difference between repartition and coalesce in Apache Spark?

Write an SQL query to find the second-highest salary from an employee table.

What is the difference between cache() and persist() in Spark? When would you use each?

What is the difference between groupByKey and reduceByKey in Spark?

What is the difference between narrow and wide transformations in Apache Spark? Explain with examples.

What architecture are you following in your current project, and why?

Demonstrate the difference between DENSE_RANK() and RANK()

Explain the differences between Data Warehouse, Data Lake, and Delta Lake

Explain the differences between Repartition and Coalesce. When would you use each?

What is the difference between partitioning and bucketing in Spark, and when would you use bucketing?

What strategies can you use to handle skewed data in Spark?

Briefly introduce yourself and walk us through your journey as a Data Engineer so far.

Describe a scenario where partitioning and bucketing would improve query performance.

Explain the types of triggers in ADF, including schedule, tumbling window, and event-based triggers.

How do you remove duplicate rows in BigQuery?

Joins and window functions - INNER, LEFT, RIGHT, FULL OUTER, ROW_NUMBER(), RANK(), DENSE_RANK()

Can you explain the architecture of Apache Spark and its components?

Describe the difference between Spark RDDs, DataFrames, and Datasets.

Explain the difference between Spark's map() and flatMap() transformations.