Multiprocessing in Python - explain with example

Why Multiprocessing vs Threading: GIL limits threading to single-core CPU for CPU-bound work. Multiprocessing spawns separate processes—true parallelism, each with own GIL. Use for: CPU-bound (numeric, parsing), NOT for I/O-bound (use asyncio/threading).

Scalability Trade-offs: (1) Process spawn overhead: ~50–100ms each—amortize over chunk size. (2) IPC: Queue/Pipe have serialization cost; shared memory (multiprocessing.Value) for large arrays. (3) Pool.map: simple but materializes all results—use imap for streaming. (4) On Windows: if __name__ == '__main__' is mandatory (spawn semantics).

Cost: 4 workers = 4x memory. For data pipelines: partition data, process in parallel, avoid sending huge objects between processes.

from multiprocessing import Pool
with Pool(4) as pool:
results = pool.map(process_chunk, chunks)