How do you handle passing parameters between notebooks?

Parameters can be passed between notebooks using environment-specific utilities like Databricks widgets, programmatic execution tools like Papermill, or external orchestrators such as Airflow. For more complex data, shared storage like files or tables can also serve as an intermediary.

Methods for Parameter Passing

* Notebook-Specific Utilities (e.g., Databricks Widgets): Databricks offers dbutils.widgets for creating interactive input fields or defining job parameters. These are retrieved using dbutils.widgets.get("param_name"). For direct notebook-to-notebook execution, the %run magic command passes parameters as environment variables, e.g., %run ./notebook $param=value, which the target notebook accesses. This is ideal for interactive development and simple job parameterization within Databricks.

* Programmatic Execution Tools (e.g., Papermill): Papermill enables parameterizing and executing notebooks programmatically, often used in automated pipelines. It injects parameters into a notebook and executes it, saving the output. For example: papermill input.ipynb output.ipynb -p param_name value. This provides reproducible execution and parameterization independent of the notebook environment.

* Shared State (Cloud Storage or Tables): For passing larger datasets, complex objects, or state that persists across different jobs, writing to a shared location is effective. This could be a file on cloud storage (e.g., DBFS, S3, ADLS) or a table in a data warehouse (e.g., a Delta Lake table). The producing notebook writes the data, and the consuming notebook reads it. This method is suitable when parameters are themselves data or require persistence.

* Orchestrators (e.g., Airflow, Azure Data Factory): Workflow orchestrators are designed to manage dependencies and pass parameters between tasks, including notebooks. Airflow operators (e.g., DatabricksSubmitRunOperator) can accept parameters, often using Jinja templating, which are then passed to the notebook task. This centralizes parameter management and ensures consistent execution across complex workflows.

Key Trade-offs and Example:
Choosing the right method depends on interactivity needs, data volume, and orchestration complexity. For interactive Databricks development, widgets are convenient. For automated, reproducible execution, Papermill or an orchestrator are preferred. Shared state is best for data-intensive parameter passing or persistent state.

# Databricks widget example in the receiving notebook
dbutils.widgets.text("report_date", "2023-01-01", "Report Date")
report_date = dbutils.widgets.get("report_date")
print(f"Generating report for: {report_date}")

In the interview, also mention:
Always document parameters, validate their inputs for correctness and security, and provide sensible default values for local development to ensure robustness and ease of use.