Write a PySpark script to read a CSV file, filter rows where the age column is less than 18, and write the result to a new CSV file.

Why It Matters (Architectural Logic): CSV is ubiquitous but inefficient at scale. Filter early; prefer Parquet for production. Handle nulls and type coercion.

Read CSV with schema inference or explicit schema: df = spark.read.option("header", "true").csv("s3://bucket/input.csv"). Filter: minors_df = df.filter(F.col("age") < 18). If age may be string: df.filter(F.col("age").cast("int") < 18). Write: minors_df.write.option("header", "true").mode("overwrite").csv("s3://bucket/output.csv"). Best practices: use repartition(1) for a single output file if needed; use coalesce for controlled parallelism; consider Parquet over CSV for performance; set escape/delimiter options for complex CSVs. Handle nulls in age: df.filter(F.col("age").isNotNull() & (F.col("age").cast("int") < 18)).

Scalability Trade-offs: CSV has no predicate pushdown—full scan. Provide schema to skip inference. repartition(1) for single file risks OOM on large data.

Cost Implications: Parquet 2-5x smaller and faster. Use CSV for small, ad-hoc; Parquet for production.