Priority Queue Problem - task prioritization and dynamic sorting

Why Priority Queues in Data Pipelines: Task scheduling (Airflow, Celery), event processing (process by timestamp/priority), and merge of K sorted streams (heapq.merge).

Implementations: (1) heapq: min-heap, O(log n) push/pop. (2) queue.PriorityQueue: thread-safe wrapper. (3) For dynamic priority updates: need decrease_key—heapq doesn't support; use lazy deletion (mark stale, ignore when popped) or custom Fibonacci heap.

Scalability: Single-process heap = millions of items. For distributed: use external queue (SQS, RabbitMQ) with priority. Celery uses Redis/RabbitMQ for priority queues—workers pull by priority.

import heapq
tasks = []
heapq.heappush(tasks, (priority, counter, task))
task = heapq.heappop(tasks) # lowest priority first