Bloom Filters in Spark projects - explain use case

Why Bloom filters matter: Probabilistic set membership—O(1) lookup, no false negatives, tunable false positive rate. Use case: Large-fact/small-dimension join—build Bloom filter of dimension keys, broadcast to executors; prune fact partitions before shuffle. Example: 1B fact rows, 10M dimension—Bloom filter of 10M keys ~10MB; most fact partitions prune early. Scalability trade-offs: False positives grow with size; tune FPR (e.g., 1%) vs memory. Parquet row-group Bloom filters (if present) enable file-level pruning. Cost implications: Reduces shuffle bytes dramatically when match rate is low—often 50–90% cost reduction for sparse joins. Best practice: Use for join optimization; measure FPR vs shuffle savings.