Reviewed by Aditya Kumar · Last reviewed 2026-03-24
Bloom filter: Probabilistic set membership; no false negatives; tunable false positive rate. Why in Spark: Dynamic partition pruning—filter on one side, build bloom filter, push to other side to skip partitions/rows; reduces I/O. Architectural Logic: Effective when filter has...
This hard-level SQL question appears frequently in data engineering interviews at companies like American Express. While less common, it tests deeper understanding that distinguishes strong candidates. Mastering the underlying concepts (join, optimization, partition) will help you answer variations of this question confidently.
This is a senior-level question that tests architectural thinking. Lead with the high-level design, then drill into specifics. Discuss trade-offs explicitly - there is rarely one correct answer. Show awareness of scale, fault tolerance, and operational complexity.
Bloom filter: Probabilistic set membership; no false negatives; tunable false positive rate. Why in Spark: Dynamic partition pruning—filter on one side, build bloom filter, push to other side to skip partitions/rows; reduces I/O. Architectural Logic: Effective when filter has high selectivity; dimension keys applied to fact table partitions. False positive impact: Over-inclusion means extra rows scanned; tunable via bits-per-element. Scalability: Bloom filter build adds CPU; at petabyte scale, filter distribution can add network. Cost: Fewer scans = lower S3/network cost; filter build = driver/executor CPU. Enable: spark.sql.optimizer.dynamicPartitionPruning.enabled; part of AQE. Best practice: Monitor false positive rate; use for broadcast-join optimization; ensure dimension is reasonably small.
Red Flag: Enabling bloom filters when dimension has low selectivity—filter almost never prunes, adding overhead. Pro-Move: Profile filter selectivity; for high-cardinality dimensions, consider partitioned bloom filters per partition key.
Some links below are affiliate links. If you buy through them we may earn a small commission at no extra cost to you — it helps keep DataEngPrep free.
According to DataEngPrep.tech, this is one of the most frequently asked SQL interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.