Explain the architectural rationale for using LeftAntiJoin vs. NOT IN vs. NOT EXISTS in a distributed context. When does LeftAntiJoin become a performance or scalability bottleneck, and how do broadcast vs. shuffle joins affect cost?

Why LeftAntiJoin: Declarative, optimized for set-difference; Catalyst can push predicates and choose join strategy. vs. NOT IN: Null-handling pitfalls (NULL in subquery yields no rows); often materializes subquery. vs. NOT EXISTS: Similar semantics; optimizer may rewrite to anti-join. Architectural choice: LeftAntiJoin when right side is small → broadcast; when large → shuffle hash/sort-merge. Scalability: Broadcast with large right side causes driver OOM; shuffle anti-join can be expensive if right has high cardinality and skew. Cost: Shuffle anti-joins move data across network; broadcast avoids shuffle but costs driver memory. Best practice: Profile both sides; use broadcast hint when right is small; for delta sync, consider partitioned anti-join. Example: df1.join(df2, df1.key == df2.key, "left_anti") with broadcast(df2) when df2 is tiny.