How would you handle a schema change when new files arrive?

When new files arrive with schema changes, the primary approach involves prioritizing additive schema evolution, validating incoming data against an expected schema, and leveraging robust data lake table formats or schema registries to manage and enforce these changes seamlessly.

Schema Evolution and Validation

The most common strategy is additive schema evolution, where new columns are added, and existing ones are never removed or re-typed incompatibly. For older records lacking new columns, default values (often NULL) are assigned, ensuring backward compatibility. This maintains a consistent view for existing consumers. Data validation is crucial. Incoming data should be checked against the current schema. Records or files with unexpected columns, incorrect data types, or missing mandatory fields should be quarantined, rejected, or routed to a dead-letter queue to maintain data quality and prevent downstream failures.

Tools and Techniques

Schema Registries (e.g., Confluent Schema Registry for Kafka) act as central repositories for schema definitions, enforcing contracts between data producers and consumers. This is vital in event-driven architectures where producers register schemas and consumers retrieve them, ensuring compatibility and managing schema versions. For batch processing in data lakes, modern table formats like Delta Lake, Apache Iceberg, or Apache Hudi provide built-in schema evolution capabilities. They manage schema changes through a transaction log, allowing operations like adding columns or changing column comments without rewriting entire datasets. For example, in PySpark with Delta Lake, the mergeSchema=True option automatically adds new columns from incoming data to the table schema.

df.write.format("delta") \
  .mode("append") \
  .option("mergeSchema", "true") \
  .save("/path/to/delta_table")

This approach ensures that consumers can adapt to evolving schemas. For significant, breaking changes (e.g., removing a column), a new version of the dataset might be published, requiring consumers to explicitly migrate or be updated to handle the new structure.

In the interview, also mention: The trade-off between strict schema enforcement (higher data quality, less flexibility) and schema evolution (more flexibility, potential for data quality issues if not managed well).