How do you handle memory management in Python?

Python manages memory primarily through reference counting, which automatically frees objects when no longer referenced, supplemented by a generational garbage collector to detect and break reference cycles. PyMalloc optimizes allocation for small objects.

Core Mechanisms

Python's memory management relies on several integrated mechanisms:

* Reference Counting: The primary mechanism. Each Python object maintains an integer count of references pointing to it. When an object's reference count drops to zero, it's immediately deallocated, and its memory is returned to the free list. This is efficient for most cases, providing deterministic deallocation.
* Generational Garbage Collector (GC): Handles circular references, which reference counting alone cannot resolve (e.g., object A references B, B references A). The GC is generational, meaning it categorizes objects into "generations" based on their age, collecting younger generations more frequently as they are more likely to contain short-lived objects. It uses a mark-and-sweep algorithm to identify and reclaim unreachable objects involved in cycles.
* PyMalloc: An arena-based allocator specifically for small Python objects (typically less than 512 bytes). It pre-allocates large blocks of memory from the operating system and manages smaller chunks within them, reducing overhead and fragmentation for common Python objects like integers, strings, and tuples.

Why it Matters for Data Engineering

Effective memory management is critical for data engineering workflows, which often involve processing large datasets and long-running services. Memory leaks, often caused by unclosed resources or undetected reference cycles, can lead to:

* Out-of-Memory (OOM) errors: Crashing long-running ETL jobs, streaming applications (e.g., Kafka consumers), or API services.
* Performance degradation: Excessive paging to disk, slowing down processing in systems like Spark or dbt.
* Increased cloud costs: Over-provisioning compute resources (e.g., larger EC2 instances, more Spark executors) to compensate for inefficient memory usage.
* Unstable pipelines: Intermittent failures and difficult-to-diagnose issues in production environments.

Best Practices & Trade-offs

* Use Generators/Iterators: For processing large data streams (e.g., reading large files, database results), generators yield items one by one, keeping only a small portion in memory at any time. This is fundamental for scaling data pipelines.

    def read_large_file(filepath):
        with open(filepath, 'r') as f:
            for line in f:
                yield line.strip()
    

* Context Managers (with statement): Ensure resources like files, network connections, and database cursors are properly acquired and released, preventing resource leaks.
* Avoid Circular References: While the GC handles them, explicitly breaking cycles (e.g., setting references to None) can reduce GC overhead and improve performance.
* Profiling: Use tools like tracemalloc or memory_profiler to identify memory hotspots and leaks in your code.
* Tune GC: For specific long-running applications, gc.set_threshold() can adjust when the GC runs, balancing memory footprint against potential pause times.

In the interview, also mention: how del only decrements an object's reference count; it doesn't guarantee immediate deallocation unless that count drops to zero.