How do you clean missing values in a pandas DataFrame?

To clean missing values in a pandas DataFrame, the primary methods are df.dropna() for removing rows or columns, and df.fillna() for imputing values. The choice depends on the nature of the missingness, the data type, and the impact on downstream analysis.

Mechanics and Why

Before cleaning, it's crucial to identify missing values using df.isna() or df.isnull() and understand their distribution (df.isnull().sum()).

* df.dropna(): This method removes rows or columns containing missing values. Parameters like how='any' (default) or how='all' and thresh (minimum non-null values required) provide control over the removal criteria. Use dropna() when the amount of missing data is small, or when the missingness is truly random and dropping won't introduce significant bias or data loss.
* df.fillna(): This method imputes missing values. Common strategies include:
* Statistical Imputation: For numerical columns, fillna(df['column'].mean()) or fillna(df['column'].median()) are common to preserve central tendency. For categorical columns, fillna(df['column'].mode()[0]) uses the most frequent category.
* Forward/Backward Fill: ffill() (forward fill) or bfill() (backward fill) are effective for time-series or ordered data, propagating the last known valid observation.
* Constant Value: fillna(0) or another specific value can be used when a default or placeholder is appropriate.
* Per-Column Strategy: It's often best to apply different fillna() strategies per column based on data type and domain knowledge.

Documenting the chosen cleaning strategy is vital for data governance and reproducibility, especially in production pipelines (e.g., within dbt models) to ensure data lineage and consistency.

Key Trade-offs and Production Considerations

While dropna() is simple, it can lead to significant data loss and introduce bias if missingness is not completely random. Imputation with fillna() can preserve more data but risks distorting the original data distribution or introducing artificial patterns.

In production, it's critical to validate that imputation doesn't bias downstream models or analytics. This involves comparing key statistics (mean, median, standard deviation) and distributions (histograms) before and after imputation. For example, in a Spark or Snowflake environment, these data quality checks would be integrated into ETL/ELT stages before data is consumed.

import pandas as pd
import numpy as np
df = pd.DataFrame({
    'A': [1, 2, np.nan, 4],
    'B': [np.nan, 6, 7, 8],
    'C': ['X', 'Y', 'X', np.nan]
})
# Impute 'A' with median, 'B' with mean, 'C' with mode
df['A'] = df['A'].fillna(df['A'].median())
df['B'] = df['B'].fillna(df['B'].mean())
df['C'] = df['C'].fillna(df['C'].mode()[0])

In the interview, also mention: The importance of understanding the mechanism of missing data (Missing Completely At Random - MCAR, Missing At Random - MAR, Missing Not At Random - MNAR) as this informs the most appropriate cleaning strategy and potential biases.