Basic Spark commands – Create RDD, Load data, Filter

Why syntax + semantics matter: RDD vs DataFrame choice affects optimization; load options impact parallelism. Commands: Create RDD: sc.parallelize([1,2,3]) or sc.textFile("path"). Load: textFile, wholeTextFiles, sequenceFile (RDD); spark.read.csv(), spark.read.parquet() (DataFrame). Filter: rdd.filter(lambda x: x > 0); df.filter(col("status") == "active"). Scalability trade-offs: textFile partitions by block; wholeTextFiles one partition per file—bad for many small files. DataFrame with schema avoids schema inference cost on large reads. Cost implications: Inferring schema on 1TB = expensive scan; provide schema for production. Best practice: Prefer DataFrame; use schema; leverage columnar formats.