Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Transform id, dob, name → age, firstname, lastname: `from pyspark.sql.functions import datediff, current_date, split`;...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Transform id, dob, name → age, firstname, lastname: from pyspark.sql.functions import datediff, current_date, split; df.withColumn('age', datediff(current_date(), col('dob'))/365). df.withColumn('name_split', split(col('name'),' ')).withColumn('firstname', col('name_split')[0]).withColumn('lastname', col('name_split')[1]). Best practice: Handle multi-part names; validate DOB.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Split without handling multi-part. Pro-Move: 'Handle names; validate DOB.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.