Real questions from top companies · hard
Tell me about yourself and your experience.
What is the difference between SparkSession and SparkContext in Spark?
What architecture are you following in your current project, and why?
Briefly introduce yourself and walk us through your journey as a Data Engineer so far.
What is a Common Table Expression (CTE), and when would you use it?
What is the difference between a primary key and a unique key?
Explain Fact and Dimension Tables with examples.
Joins and window functions - INNER, LEFT, RIGHT, FULL OUTER, ROW_NUMBER(), RANK(), DENSE_RANK()
Can you explain the architecture of Apache Spark and its components?
Describe the difference between Spark RDDs, DataFrames, and Datasets.
How does Spark's Catalyst Optimizer work? Explain its stages.
How do you handle late-arriving data in Spark Structured Streaming?
What is the small-file problem in Spark, and how do you solve it?
How do you optimize Spark jobs for better performance? Mention at least 5 techniques.
How do you handle conflicts within a team? Provide an example.
Why are you leaving your current company?
What are the key components of AWS Glue, and how do they work together?
What is Snowflake's architecture, and why is it unique?
What is the difference between S3 and HDFS?
Briefly explain the architecture of Kafka.