Write a higher-order function to filter values greater than a threshold in a list.

A higher-order function either accepts one or more functions as arguments or returns a function as its result. In this context, filter_above is a higher-order function because it generates and returns a new filtering function (a lambda expression) tailored to a specific threshold.

Mechanics and Why

The filter_above function takes a threshold and returns a lambda function lambda x: x > threshold. This returned lambda is a predicate that can then be passed to Python's built-in filter() function, which is itself a higher-order function. filter() iterates through a sequence, applying the predicate to each element and yielding only those for which the predicate returns True. This pattern promotes reusability and composability, allowing you to easily create various filtering predicates without rewriting the core logic. For instance, you could pipe multiple such filters together. functools.partial offers an alternative way to create specialized versions of functions by "pre-filling" some of their arguments, effectively achieving a similar goal of creating a new function from an existing one.

Example and Real-World Application

This functional pattern is fundamental in data engineering. For example, in Apache Spark, transformations like DataFrame.filter() or RDD.filter() accept a function or an expression to define the filtering logic. Similarly, in Pandas, you often pass lambda functions or boolean conditions to filter DataFrames.

from functools import partial
def filter_above(threshold):
    """Returns a predicate function that checks if a value is greater than the threshold."""
    return lambda x: x > threshold
numbers = [5, 12, 3, 18, 9, 22]
# Using filter_above to create a predicate
is_above_10 = filter_above(10)
filtered_numbers_1 = list(filter(is_above_10, numbers))
# Result: [12, 18, 22]
# Direct usage
filtered_numbers_2 = list(filter(filter_above(15), numbers))
# Result: [18, 22]
# Using functools.partial to create a specialized filter operation
# This creates a function that, when called with a list, filters it.
filter_gt10_operation = partial(filter, lambda x: x > 10)
filtered_numbers_3 = list(filter_gt10_operation(numbers))
# Result: [12, 18, 22]

In the interview, also mention…

Emphasize that such higher-order functions often promote immutability and pure functions (no side effects), making code easier to test, debug, and reason about, especially in distributed data processing systems.