How do you interact with Google BigQuery using Python?

You interact with Google BigQuery using Python primarily through the google-cloud-bigquery client library. This library provides a Client object to manage connections and execute various operations, from querying to data loading.

After importing google.cloud.bigquery and instantiating client = bigquery.Client(), you execute SQL queries using client.query("SELECT * FROM dataset.table"). The method returns a QueryJob object; call .result() to wait for completion and retrieve rows. For convenience, .to_dataframe() converts results into a Pandas DataFrame, suitable for analysis.

For secure and efficient queries, especially in production, use parameterized queries via QueryJobConfig(query_parameters=[...]). This prevents SQL injection and allows BigQuery to cache query plans, improving performance.

The library supports various data loading methods: client.load_table_from_dataframe(df, 'project.dataset.table') for in-memory data, and client.load_table_from_uri('gs://bucket/file.csv', 'project.dataset.table') for external data from Cloud Storage, which is ideal for large volumes.

Crucially, QueryJobConfig(dry_run=True) allows you to validate query syntax and estimate bytes scanned before execution, directly impacting cost. After a job completes, inspect job.total_bytes_processed or job.total_bytes_billed for actual cost metrics. Authentication defaults to Application Default Credentials (ADC), which automatically finds credentials from the environment (e.g., GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to a service account JSON key, or GCE metadata).

Here's an example of a parameterized query with a dry run:

from google.cloud import bigquery
client = bigquery.Client()
query = "SELECT name, age FROM project.dataset.table WHERE age > @min_age"
job_config = bigquery.QueryJobConfig(
    dry_run=True,
    query_parameters=[
        bigquery.ScalarQueryParameter("min_age", "INT64", 30),
    ]
)
query_job = client.query(query, job_config=job_config)
print(f"Estimated bytes scanned: {query_job.total_bytes_processed}")

In the interview, also mention best practices like robust error handling, retries for transient failures, and considering asynchronous job management for long-running operations.