Q: Garbage Collector in Python - explain

**Python GC:** Ref counting + generational for cycles. **Module:** gc.collect(), get_count(). **CPython:** Refcount primary. **Cycles:** Generations. **Why:** Rare tuning. **Production:** Avoid circular refs for large objects.

Question 1

Explain the differences between a Data Lake and a Data Warehouse.

Accepted Answer

**Data Lake**: Low-cost object storage (S3, ADLS) for raw, semi-structured, unstructured data. Schema-on-read; used for exploratory analytics, ML, archival. **Data Warehouse**: Structured, curated storage optimized for SQL; schema-on-write; used for BI and reporting. **Why both exist**: Lakes offer flexibility and cost at scale; warehouses offer query performance and concurrency....

Question 2

Docker - purpose and handling dependencies

Accepted Answer

Architectural logic: Docker containerizes app + dependencies for portability and reproducibility. Why: 'Runs on my machine' → runs everywhere; isolate from host. Dependencies: Baked into image via Dockerfile (FROM, RUN pip install, COPY). Scalability: Same image scales horizontally; registry for distribution. Cost: Smaller images = faster pull, less storage....

Question 3

Libraries for Data Wrangling

Accepted Answer

Python: Pandas (tabular), Polars (faster, lazy), PySpark (distributed). R: dplyr, data.table. ETL: Pandas for <10GB; Polars for single-node performance; PySpark for big data. WHY: Different scale and latency requirements drive choice. SCALABILITY: Pandas hits memory limits; Polars uses Apache Arrow and multithreading; Spark scales horizontally. COST: Polars is free and often 5–10x faster than Pandas—reduces compute time; Spark clusters have fixed cost....

Question 4

Garbage Collector in Python - explain

Accepted Answer

**Python GC:** Ref counting + generational for cycles. **Module:** gc.collect(), get_count(). **CPython:** Refcount primary. **Cycles:** Generations. **Why:** Rare tuning. **Production:** Avoid circular refs for large objects.

Question 5

GeoPandas - definition and features

Accepted Answer

**GeoPandas:** Pandas + geospatial. GeoDataFrame; geometry column. Shapefiles, GeoJSON. Spatial joins, overlays, buffering. Shapely, pyproj. **Use:** Maps; spatial analytics; routing. **Why:** Unified tabular+spatial....

Question 6

Multiprocessing in Python - explain with example

Accepted Answer

**Why Multiprocessing vs Threading:** GIL limits threading to single-core CPU for CPU-bound work. Multiprocessing spawns separate processes—true parallelism, each with own GIL. Use for: CPU-bound (numeric, parsing), NOT for I/O-bound (use asyncio/threading).

**Scalability Trade-offs:** (1) Process spawn overhead: ~50–100ms each—amortize over chunk size. (2) IPC: Queue/Pipe have serialization cost; shared memory (multiprocessing.Value) for large arrays....

Question 7

Stuff Function for XML Usages

Accepted Answer

**Why STUFF + FOR XML PATH:** SQL Server pattern for string aggregation pre-2017. Builds comma-separated lists, XML fragments. Replaced by STRING_AGG in modern SQL.

**Syntax:** STUFF(string, start, length, replace)—replaces chars. Combined with FOR XML PATH(''): concatenates rows into string; STUFF removes leading comma.

**Scalability:** FOR XML PATH scans all rows—O(n). For large aggregations (100K+ rows per group), consider CLR aggregate or application-layer concatenation....

Question 8

Indexing: When to Use and Avoid

Accepted Answer

Use indexes when: high selectivity (many distinct values), frequent filters/joins, read-heavy workload. Avoid when: low cardinality (e.g., gender), small tables (full scan is cheap), write-heavy (index maintenance cost), columns rarely in WHERE. Over-indexing causes slow writes and storage bloat. Rule of thumb: index columns in WHERE, JOIN ON, ORDER BY. Monitor with pg_stat_user_indexes (Postgres) or sys.dm_db_index_usage_stats (SQL Server). Drop unused indexes....

NAB Data Engineer Interview Questions

+4 More Questions with Expert Answers

Other Companies

NAB Data Engineer Interview Questions

+4 More Questions with Expert Answers

Other Companies