Reviewed by Aditya Kumar · Last reviewed 2026-03-24
**Why Streaming:** Loading 10GB JSON kills memory. Use JSON Lines (ndjson)—one JSON object per line; iterate line-by-line. For nested: ijson for streaming parse. **Pattern:** Open file, for line in f: obj=json.loads(line); if cond: batch.append(obj). Batch insert (executemany)...
Why Streaming: Loading 10GB JSON kills memory. Use JSON Lines (ndjson)—one JSON object per line; iterate line-by-line. For nested: ijson for streaming parse.
Pattern: Open file, for line in f: obj=json.loads(line); if cond: batch.append(obj). Batch insert (executemany) every 1000 rows. Commit in batches.
Production: Use connection pooling. Progress logging. Handle malformed lines (try/except, dead-letter). At Goldman: schema validation before insert.
for line in open(path):
obj = json.loads(line)
if cond(obj): batch.append(obj)
if len(batch) >= 1000: cursor.executemany(insert_sql, batch); batch.clear()
Red Flag: json.load() on large file. Pro-Move: 'We use ijson for nested; JSON Lines + batches of 5K for our 50GB event files.'
According to DataEngPrep.tech, this is one of the most frequently asked Python/Coding interview questions, reported at 1 company. DataEngPrep.tech maintains an editor-reviewed database of 1,863 data engineering interview questions across 7 categories.