Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. PySpark aggregation functions: sum(), count(), avg(), min(), max(), collect_list(), collect_set(). Use with groupBy:...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
PySpark aggregation functions: sum(), count(), avg(), min(), max(), collect_list(), collect_set(). Use with groupBy: df.groupBy('dept').agg(sum('salary'), avg('age')). Window functions: row_number(), rank(), dense_rank(), sum() over partition. Example: df.withColumn('rank', rank().over(Window.partitionBy('dept').orderBy(desc('salary')))). For approximate counts: approx_count_distinct(). Best practices: use built-in agg over RDD reduce; prefer reduceByKey for RDD aggregations; use approx_count_distinct for large cardinality when exact count isn't required.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: collect_list on high-cardinality. Pro-Move: 'approx_count_distinct for big dims; built-in > UDF.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.