#partition

Write a SQL query to find top 3 earners in each department.

Write a query to find the top three highest-paid employees in each department using window functions.

Write complex SQL queries involving multiple joins, subqueries, and data aggregation logic.

Convert complex SQL (CTEs, window functions, subqueries) to production-grade PySpark. Discuss when to use spark.sql() vs. DataFrame API, and the implications for testability, partitioning, and execution predictability.

Explain how Adaptive Query Execution changes the economics of Spark tuning. What problems does it solve at runtime, and when might you still need manual intervention (e.g., salting, broadcast hints)?

Architect incremental load in ADF + Databricks with idempotency, late-arrival handling, and cost/scalability implications of watermark vs. change data capture.

Explain strategies for managing schema changes in PySpark over time.

How do you drop columns with null values in PySpark?