Implement a Spark job to find the top 10 most frequent words in a large text file.

Core logic: read text → split → explode → filter empty → groupBy → count → orderBy desc → limit 10. Code: from pyspark.sql import functions as F; df = spark.read.text("path/to/file.txt"); words = df.select(F.explode(F.split(F.col("value"), "\\s+")).alias("word")); top10 = words.filter(F.length(F.col("word")) > 0).groupBy("word").count().orderBy(F.desc("count")).limit(10). Why \\s+: Handles multiple spaces/tabs; more robust than single space. Scalability: For very large files, ensure enough partitions (coalesce input or repartition after read); reduceByKey equivalent is groupBy+agg. Cost: Single action (limit triggers collect); for distributed top-K without collecting to driver, use Window functions: row_number() over (partition by 1 order by count desc) and filter rank <= 10. Architectural nuance: For multi-file corpus, read as wholeTextFiles or text with glob; partition count affects parallelism. Best practice: Normalize case (lower) and strip punctuation if word identity matters.