Question 1

Can you give an example of processing nested JSON data using these functions?

Accepted Answer

**Spark:** `from_json` with schema; `explode` for arrays; struct access. **Pandas:** `json_normalize`. **Example:** `df.select(col('data').getItem('nested').getItem('field'))`. **Why:** Explicit schemas; handle missing keys. **Production:** `schema_of_json` for inference; flatten for analytics; retain nested for storage....

Question 2

How do you install a Python library that is not in the Databricks runtime?

Accepted Answer

**Options:** (1) %pip in notebook. (2) Cluster library—UI or API. (3) Init script: dbutils.library.installPyPI. (4) Wheel on DBFS. **Scoped:** Cluster vs notebook. **Production:** Cluster libs; version pinning....

Question 3

When would you use flatten, explode, or collect_list in Spark?

Accepted Answer

**Semantics:** explode = one-to-many (array → rows). flatten = array of arrays → single array (no new rows). collect_list = many-to-one (aggregate to array per group).

**Use Cases:** explode: normalize nested JSON. flatten: merge nested array before processing. collect_list: group by key, collect values (e.g., order items per order_id).

**Scalability:** explode multiplies rows—risk of skew. collect_list can create huge arrays—monitor group sizes....

TCS Python Interview Questions

Companies asking these questions

All 3 Questions

More Interview Prep Guides

Get All Answers in PDF Format

TCS Python Interview Questions

Companies asking these questions

All 3 Questions

More Interview Prep Guides

Get All Answers in PDF Format