Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Lead/Lag in PySpark: from pyspark.sql.window import Window; from pyspark.sql.functions import lead, lag; w = Window.partitionBy('customer_id').orderBy('order_date'); df.withColumn('next_order', lead('order_id', 1).over(w)).withColumn('prev_order', lag('order_id', 1).over(w))....
Lead/Lag in PySpark: from pyspark.sql.window import Window; from pyspark.sql.functions import lead, lag; w = Window.partitionBy('customer_id').orderBy('order_date'); df.withColumn('next_order', lead('order_id', 1).over(w)).withColumn('prev_order', lag('order_id', 1).over(w)). Lead: next row; Lag: previous row. Default for no row: None. Use lead('col', 1, 0) for default. For SQL: SELECT *, LAG(amount) OVER (PARTITION BY id ORDER BY dt) prev, LEAD(amount) OVER (...) next FROM t. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.
Red Flag: Arbitrary partition counts. Pro-Move: 'Partitions = size/150MB; we monitored Spark UI for skew, adjusted when P99 > 2× median.'
According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.