Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Broadcast join + conditional agg: `from pyspark.sql.functions import broadcast, sum, when`; `joined = df.join(broadcast(lookup_df),...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Broadcast join + conditional agg: from pyspark.sql.functions import broadcast, sum, when; joined = df.join(broadcast(lookup_df), 'id'); joined.groupBy('location').agg(sum(when(col('status')=='A',1).otherwise(0)).alias('count_a')). Best practice: Broadcast < 10MB; use for dimension tables.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Broadcast > 10MB. Pro-Move: 'Broadcast small; when/otherwise agg.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.