How do you drop columns with null values in PySpark?

Two distinct operations: (1) Drop columns that are entirely null (no non-null values): null_cols = [c for c in df.columns if df.filter(col(c).isNotNull()).count() == 0]; df = df.drop(*null_cols). Caveat: count() triggers a full scan—expensive on large tables. (2) Drop rows with null in specified columns: df.dropna(subset=["col1", "col2"]). Scalability: The column-null check is O(partitions × columns) and can be costly; consider sampling or inferring from schema/sample. Production logic: Log dropped columns for audit; use schema evolution if columns appear conditionally (e.g., A/B test variants). Why not drop all null columns blindly: Some columns are legitimately sparse (e.g., optional fields); dropping removes signal. Best practice: Define critical vs. optional columns in config; drop only optional all-null columns; validate with data quality checks.