Command to Read JSON Data and Options

To read JSON data in Spark, the primary commands are spark.read.json('path/to/data.json') or the more explicit spark.read.format('json').load('path/to/data.json'). These methods create a DataFrame by parsing JSON files from a specified path.

Mechanics and Options

By default, Spark performs schema inference by sampling a portion of the JSON data. While convenient for exploration, this can be slow for large datasets and prone to errors if the sample isn't representative, leading to incorrect data types or nulls. For production workloads, defining an explicit schema using StructType is a best practice. This avoids the inference pass, improves performance, and ensures data type consistency.

Key options include:
* schema: A pyspark.sql.types.StructType object to explicitly define the DataFrame's structure, crucial for performance and data quality.
* multiLine: Set to true when reading pretty-printed JSON files where a single record spans multiple lines. Be cautious, as this requires Spark to read the entire file into memory for parsing, potentially leading to OutOfMemory errors on very large files.
* dateFormat / timestampFormat: Specifies custom date/timestamp patterns for parsing string representations into DateType or TimestampType.
* primitivesAsString: Reads all primitive values as strings, useful when dealing with ambiguous or mixed-type fields.
* mode: Controls how Spark handles malformed records (e.g., PERMISSIVE (default), DROPMALFORMED, FAILFAST).

For complex or semi-structured JSON embedded within a column (e.g., a string column containing JSON), the from_json function (from pyspark.sql.functions) is used with a predefined schema to parse the string into a StructType column. For streaming JSON, spark.readStream.schema(schema).json(path) is used, where an explicit schema is mandatory.

Example and Best Practices

Defining a schema is critical. It allows Spark to optimize reads through column pruning, fetching only the necessary fields from storage. Leveraging a Glue Catalog or similar metadata store to manage and infer schemas for your data lakes ensures consistent access and data governance across different Spark applications.

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, TimestampType
# Define an explicit schema for robustness and performance
json_schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("name", StringType(), True),
    StructField("timestamp", TimestampType(), True)
])
df = spark.read \
    .schema(json_schema) \
    .option("multiLine", "false") \
    .option("timestampFormat", "yyyy-MM-dd HH:mm:ss") \
    .json("s3://your-bucket/data.json")
df.printSchema()

In the interview, also mention strategies for handling schema evolution (e.g., using Delta Lake or Iceberg) and robust error handling for malformed records, such as directing them to a badRecordsPath for later analysis.