How do you handle exceptions in Python? Provide an example.

Python handles exceptions using the try, except, else, and finally blocks. This mechanism is crucial for building robust data pipelines, preventing crashes, and ensuring critical operations like resource cleanup always execute.

Mechanics and Purpose

* try: This block contains the code that might raise an exception.
* except: If an exception occurs within the try block, the corresponding except block (matching the exception type) is executed. You can have multiple except blocks to handle different types of errors specifically.
* else: This optional block executes only if the try block completes successfully without raising any exceptions. It's useful for code that should run exclusively upon successful execution.
* finally: This block guarantees execution regardless of whether an exception occurred, was handled, or even if the try block completed successfully. It is essential for deterministic cleanup operations, such as closing file handles, database connections, or network sockets, preventing resource leaks that can impact scalability and cost in data systems.

Example and Best Practices

Consider a data engineering task involving reading a configuration file.

import logging
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
def load_config(filepath):
    file_handle = None
    try:
        file_handle = open(filepath, 'r')
        config_data = file_handle.read()
        logging.info(f"Configuration loaded successfully from {filepath}")
        return config_data
    except FileNotFoundError:
        logging.error(f"Config file not found: {filepath}. Please ensure it exists.")
        raise # Re-raise to signal a critical failure upstream
    except IOError as e:
        logging.error(f"Error reading config file {filepath}: {e}. Check permissions.")
        # Potentially attempt a retry or use a default config here
    except Exception as e:
        logging.critical(f"An unexpected error occurred loading config {filepath}: {e}")
        raise # Catch-all for unknown issues, re-raise as a last resort
    finally:
        if file_handle:
            file_handle.close()
            logging.debug(f"Closed file handle for {filepath}")

Key Best Practices:

Specific Exceptions: Always catch specific exception types (e.g., FileNotFoundError, ZeroDivisionError). A bare except: (without specifying an exception type) is dangerous in production as it catches all* exceptions, including KeyboardInterrupt or SystemExit, masking critical issues and preventing graceful shutdowns.
* Logging with Context: Log exceptions with sufficient context (e.g., file path, record ID, stack trace) to aid debugging. Poor logging can mask bugs, leading to corrupted data or resource exhaustion in distributed systems like Spark or Kafka.
* Re-raising: Use raise within an except block to handle an error locally (e.g., log it) but then propagate it up the call stack for higher-level handling or to signal a pipeline failure.
* finally for Cleanup: The finally block is paramount for preventing resource leaks. In data engineering, this ensures database connections, file handles, or Kafka producer/consumer clients are properly closed, even if data processing fails.

In the interview, also mention…

Emphasize how finally is critical for preventing resource leaks in data pipelines, ensuring connections (e.g., to Snowflake, S3, Delta Lake) are closed, even if processing fails midway, which is vital for maintaining system stability and managing cloud costs.