American Express

What is the difference between SparkSession and SparkContext in Spark?

Discuss the data size challenges in your previous projects. How did you optimize storage and processing?

What are your strengths, and how do they align with the Data Engineer role?

Implement a Python function to count unique words from a file and write them to another file.

Describe a scenario where you used Databricks for real-time data processing.

Explain bloom filters in Spark: how they reduce I/O and when they introduce false positives that hurt performance. What are the scalability and cost implications of enabling dynamic partition pruning and bloom filter pushdown at petabyte scale?

Explain a scenario-based question on Spark optimization and how you would troubleshoot performance issues.

Explain repartition vs. coalesce. Which one would you use to reduce shuffle operations?