What is broadcasting in Spark, and why is it used? Can you give an example of its use?

Broadcasting sends a small table replica to every executor so joins execute locally without shuffle. Why it matters: A shuffle join moves both sides across the network; broadcast join moves only the small side once from driver to executors. Scalability trade-off: The small table must fit in executor memory; exceeding this causes OOM. As cluster size grows, more copies exist (N executors × table size), so large broadcasts waste memory. Cost implication: Avoids expensive shuffle I/O and disk spill; typical 10–100× faster for dimension–fact joins on large fact tables. Example: from pyspark.sql.functions import broadcast; df_large.join(broadcast(df_small), "key"). Use when one side is < spark.sql.autoBroadcastJoinThreshold (default 10MB). Explicit broadcast() guarantees behavior; Spark may auto-broadcast otherwise. Best practice: Check table size before broadcast; use for lookup/dimension tables in star schema; increase threshold only on small clusters with sufficient memory.