What are decorators in Python, and how do they work?

Decorators in Python are higher-order functions that allow you to modify or enhance the behavior of another function or method without explicitly altering its source code. They provide a clean, readable way to "wrap" functions, typically using the @decorator_name syntax placed directly above the function definition, making code more declarative.

How They Work

A decorator function takes the original function as an argument, defines an inner wrapper function that encapsulates the original function call (adding logic before, after, or around it), and then returns this wrapper. The @decorator_name syntax is syntactic sugar for assigning the result of decorator_name(original_function) back to original_function. This mechanism is powerful for implementing separation of concerns, allowing you to isolate cross-cutting functionalities like logging, authentication, caching, retries, or performance monitoring from the core business logic, thereby improving modularity and reusability.

For example, a common decorator in data engineering is a timer to measure execution time:

import time
import functools

def timer(func):
    @functools.wraps(func)
    def wrapper(args, *kwargs):
        start_time = time.perf_counter()
        result = func(args, *kwargs)
        end_time = time.perf_counter()
        print(f"Function {func.__name__!r} executed in {end_time - start_time:.4f}s")
        return result
    return wrapper
@timer
def fetch_data(url):
    # Simulate data fetching from an external API
    time.sleep(1)
    return f"Data from {url}"

Data Engineering Applications & Trade-offs

In data pipelines, decorators are invaluable. You might use @retry on an API client function to handle transient network errors when fetching data for a Spark job, @logged to record execution details of a dbt model's transformation step, or @rate_limited to control calls to external services like a data vendor's API. While they promote code reusability, cleaner code, and a declarative style, overuse can introduce indirection, making stack traces deeper and potentially harder to debug. They also add a slight performance overhead due to the extra function call.

In the interview, also mention functools.wraps for preserving crucial function metadata (__name__, __doc__), and consider class-based decorators for scenarios requiring stateful behavior (e.g., a rate limiter that tracks call counts over time).