Broadcast Joins and Shuffle Merge Joins?

Why join strategy matters: Shuffle is the most expensive operation; choosing wrong strategy 10x+ runtime. Broadcast join: Small table sent to all executors; no shuffle for large table. Use when one side < spark.sql.autoBroadcastJoinThreshold (~10MB default). Shuffle sort-merge: Both sides shuffled by key, sorted, merged. Default for large joins. Shuffle hash: One side hashed in memory; other streamed—faster when build side fits. Scalability trade-offs: Broadcast fails if small table grows; shuffle scales but cost = O(n). Cost implications: Broadcast = minimal network; shuffle = bytes * nodes. Tuning: increase broadcast threshold for larger dimensions (e.g., 100MB); use hints when Catalyst chooses poorly.