Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Given df1, df2: common transformations—join (df1.join(df2, 'key')), union (df1.union(df2)), concat (unionByName with...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Given df1, df2: common transformations—join (df1.join(df2, 'key')), union (df1.union(df2)), concat (unionByName with allowMissingColumns). Then apply filters, aggregates, select. Example: result = df1.join(df2, 'id').select(df1.id, df1.name, df2.score).filter(col('score')>70). Best practice: use consistent schemas; prefer join over Cartesian when possible; persist intermediate if reused.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Cartesian when join possible. Pro-Move: 'Consistent schema; broadcast small; persist reused.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.