Write a PySpark script to check for missing values and duplicate rows in a DataFrame. How would you ensure data quality before saving it to a storage system?

Why It Matters (Architectural Logic): Data quality gates prevent downstream corruption, wasted compute, and compliance issues. Systematic validation balances rigor with performance.

Data quality validation in PySpark requires systematic checks before persistence. For missing values, iterate over columns and use count() with filter(isnull()): for col in df.columns: null_count = df.filter(F.col(col).isNull()).count(); log or raise if thresholds exceeded. For duplicates: dup_count = df.count() - df.dropDuplicates().count(); use dropDuplicates() with specific columns for business-key deduplication. Best practice: define validation rules (e.g., max 5% nulls per column), run checks in a validation stage, and fail the job with clear metrics if violated. Log results to a data quality dashboard. Before saving: coalesce/repartition appropriately, use schema enforcement (e.g., Delta MERGE with schema evolution disabled for strict writes), and consider idempotent writes with overwrite by partition or merge keys.

Scalability Trade-offs: Validation scales with partition count; design checks to fail fast. Streaming validation adds latency—batch validate when possible.

Cost Implications: Catching bad data early prevents expensive reprocessing. Over-validation (full scans for uniqueness) can 2x cost—use sampling for large datasets.