Reviewed by Aditya Kumar · Last reviewed 2026-03-25
**When to Use UDFs**: Built-in functions cannot express custom business logic (e.g., proprietary scoring, external API calls, complex parsing). UDFs extend SQL/DataFrame. **Why Python UDFs Are Slow**: Each row is serialized JVM->Python->JVM; no Catalyst optimization; no...
This hard-level Spark/Big Data question appears frequently in data engineering interviews at companies like Citi. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (optimization, python, sql) will help you answer variations of this question confidently.
This is a senior-level question that tests architectural thinking. Lead with the high-level design, then drill into specifics. Discuss trade-offs explicitly - there is rarely one correct answer. Show awareness of scale, fault tolerance, and operational complexity.
When to Use UDFs: Built-in functions cannot express custom business logic (e.g., proprietary scoring, external API calls, complex parsing). UDFs extend SQL/DataFrame.
Why Python UDFs Are Slow: Each row is serialized JVM->Python->JVM; no Catalyst optimization; no vectorization. 10–100x slower than built-in.
Alternatives: (1) Pandas UDF (vectorized): Processes batches; 5–20x faster than row UDF. (2) Scala/Java UDF: No serialization overhead. (3) Built-in + expr: Often expr() or when/otherwise can replace simple UDFs.
Scalability Trade-offs: UDF breaks predicate pushdown; filter after UDF can't be pushed to source. For 1B rows, Python UDF can add hours.
Cost Implications: UDF-heavy jobs need more executors and time; 2–3x cost vs. built-in equivalent. Prefer pandas_udf with StructType for structured output.
Pro-Move: 'Replaced 3 Python UDFs with pandas_udf; job time dropped 70%.' Red Flag: Using Python UDF for simple string ops—upper(), trim() exist as built-in.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked Spark/Big Data interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.