How would you deal with data skewness in a large dataset?

Skew is when a few keys hold a disproportionate share of the rows, so one or two tasks process most of the data while the rest finish and idle. The job's runtime becomes the runtime of its slowest partition.

Diagnose before you fix

Confirm the skew rather than assuming it. In the Spark UI, look at the task duration and shuffle-read distribution for the slow stage: skew shows as a max far above the median. Then find the offending keys:

SELECT join_key, COUNT(*) AS n
FROM   events
GROUP  BY join_key
ORDER  BY n DESC
LIMIT  20;

Very often the top key is a placeholder — NULL, empty string, unknown, or a default id — that carries no business meaning and can simply be filtered out or routed around.

Enable adaptive execution first

Modern Spark handles much of this automatically by splitting oversized partitions at runtime:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

This is the cheapest fix and should be tried before hand-tuning anything.

Broadcast, if one side is small

If the skewed join has a small dimension side, broadcasting it eliminates the shuffle and therefore the skew entirely.

Salting, when the skew is genuine

When a hot key is real business data, spread it across partitions by adding a random suffix, joining on the salted key, then aggregating away the salt:

from pyspark.sql import functions as G
SALT = 10
left  = df_large.withColumn("salt", (G.rand() * SALT).cast("int"))
right = df_small.withColumn("salt", G.explode(G.array([G.lit(i) for i in range(SALT)])))
joined = left.join(right, ["join_key", "salt"])

The small side is replicated once per salt value, which is the cost of the technique.

Two-phase aggregation

For a skewed GROUP BY, aggregate by salted key first, then re-aggregate by the real key. Two cheap shuffles beat one catastrophic one.

Prevent it upstream

Partition on a higher-cardinality column, or a composite key, so the data lands balanced in the first place.

In the interview, also mention that skew is a data distribution problem, so adding executors does not help — the straggler task still runs alone.