Python for Data Engineering: Interview Questions & Answers

Python is the lingua franca of data engineering. While SQL tests your data manipulation skills, Python tests your ability to build, automate, and orchestrate. Common Python topics in DE interviews: - File handling (CSV, Parquet, JSON) - API consumption and REST client design - PySpark DataFrame operations - pandas for data transformation - Error handling and logging patterns - Unit testing data pipelines

Most Spark jobs in production are PySpark. Key patterns to know: - Reading/writing Parquet and Delta files - UDFs vs built-in functions (and why to avoid UDFs) - Broadcast variables and accumulators - Schema enforcement and evolution - Testing PySpark code with local SparkSession

Unlike SWE interviews, DE coding rounds focus on data manipulation: - Parse a nested JSON file and flatten it - Implement a simple ETL pipeline with error handling - Write a deduplication function - Build a data validation framework - Implement retry logic with exponential backoff