What is Multiline option in JSON?

The multiline option in JSON parsing, when set to true, instructs the parser to treat an entire file as a single, potentially multi-line JSON object or an array of JSON objects. This allows a single JSON record to span across multiple lines, including pretty-printed formats with indentation and line breaks.

By default, most big data processing frameworks (like Apache Spark) assume multiline=false, expecting each line in the file to represent a complete and independent JSON object. This format is known as JSON Lines (or NDJSON - Newline Delimited JSON). NDJSON is highly efficient for streaming and parallel processing because each line can be read and processed independently without needing to read the entire file.

The multiline=true option is primarily used when dealing with conventionally pretty-printed JSON files, such as those often returned by REST APIs or stored for human readability. These files typically contain a single large JSON object or an array of objects spanning many lines, often with formatting like indentation.

In PySpark, you would enable this option when reading a JSON file:

df = spark.read.option("multiline", "true").json("path/to/data.json")

The key trade-offs involve performance and memory. When multiline=true, the parser must read the entire file into memory (or a large buffer) to correctly identify and parse the single JSON object or array. This can be significantly slower and more memory-intensive, especially for very large files, as it prevents distributed processing where different parts of the file can be processed in parallel by different tasks. If the file contains a mix of single-line and multi-line objects, parsing can fail or produce unexpected results. For large-scale data ingestion and processing, NDJSON is almost always preferred due to its streamability, fault tolerance, and parallel processing capabilities.

In the interview, also mention that while multiline=true is convenient for API outputs, it's generally an anti-pattern for large-scale data lakes and warehouses, where NDJSON or more structured formats like Parquet or ORC are preferred.