Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Array to columns: from pyspark.sql.functions import col; df.select(col('array_col')[0].alias('col1'), col('array_col')[1].alias('col2'))...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Array to columns: from pyspark.sql.functions import col; df.select(col('array_col')[0].alias('col1'), col('array_col')[1].alias('col2')) or df.select(explode('array_col')).groupBy('id').pivot('array_col'). Best practice: use explode for long arrays; use inline for array of structs; avoid row-by-row when possible.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Row-by-row for array. Pro-Move: 'explode; inline for struct arrays; avoid UDF.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.