Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**Why it matters**: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams. Optimize slow Databricks notebook: (1) Cluster—right-size, enable autoscaling. (2) Data—use Delta/Parquet; filter early; avoid full...
Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.
Optimize slow Databricks notebook: (1) Cluster—right-size, enable autoscaling. (2) Data—use Delta/Parquet; filter early; avoid full scans. (3) Caching—cache reused DataFrames. (4) Parallelism—increase partitions. (5) Code—remove unnecessary cells; combine operations. (6) Use Photon if available. Best practice: Profile with Spark UI; use job clusters for scheduled runs; avoid running ad-hoc on large clusters.
Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.
Red Flag: Ad-hoc on prod cluster. Pro-Move: 'Job clusters; Photon; profile; cache.'
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.