Explain and implement Semaphore in Java

A Semaphore in Java is a counting semaphore that controls access to a shared resource by maintaining a set of "permits." It limits the number of threads that can concurrently access a critical section or a finite pool of resources.

Mechanics and Why it's Used

A semaphore is initialized with a specific number of permits (Semaphore(n)). Threads call acquire() to obtain a permit; if no permits are available, the thread blocks until one is released. After using the resource, threads call release() to return a permit. The tryAcquire() method offers a non-blocking attempt to get a permit. This mechanism provides crucial backpressure, preventing resource exhaustion (e.g., overwhelming a database or an API) and ensuring controlled access to finite resources. It's fundamental for bounding concurrency and protecting shared infrastructure within data processing applications.

Practical Application and Trade-offs

A common use case in data engineering is managing a fixed-size database connection pool or limiting concurrent calls to an external API during an ETL process. For instance, an application might use a Semaphore to ensure only a specified number of concurrent writes hit a Delta Lake table or Kafka topic, preventing performance degradation or resource contention. It's critical to always release the permit in a finally block to prevent resource leaks, even if an exception occurs. For production systems, consider fairness; new Semaphore(n, true) ensures threads acquire permits in the order they requested them, which can be vital for predictable throughput and avoiding thread starvation in long-running data pipelines.

import java.util.concurrent.Semaphore;
public class ApiRateLimiter {
    private final Semaphore semaphore;
public ApiRateLimiter(int maxConcurrentCalls) {
        this.semaphore = new Semaphore(maxConcurrentCalls);
    }
public void makeApiCall() throws InterruptedException {
        semaphore.acquire(); // Blocks if max calls reached
        try {
            // Logic to make the actual API call
            System.out.println(Thread.currentThread().getName() + " making API call.");
            Thread.sleep(100); // Simulate network latency/work
        } finally {
            semaphore.release(); // Always release the permit
        }
    }
}

In the interview, also mention its role in managing concurrent operations in distributed systems, such as limiting parallel tasks in a Spark job that interact with external services, or controlling the fan-out of data processing steps.