Your Kafka producer schema has changed, and the new data includes additional fields. How would you ensure backward compatibility using Schema Registry while consuming data from the same topic?

Why It Matters (Architectural Logic): Strict schemas reject malformed data at read time—fail fast vs. silent corruption. FAILFAST mode prevents partial loads.

Schema Registry enables schema evolution. Backward compatibility: new schema adds optional fields; old consumers ignore them. Use Avro/Protobuf with schema.registry.url. Consumer: fetch schema by id/version, deserialize. Config: auto.register.schemas=false, use.latest.version=true or use.specific.avro.reader=true. For new fields: add with defaults in Avro; use READER compatibility. Never remove required fields or change types without version bump. Test with old and new consumer versions. Production: pin schema version in critical pipelines; monitor schema registry; use separate topics for breaking changes.

Scalability Trade-offs: Schema validation is O(n) per column; parallelize across partitions. Provide schema to skip inference—3x faster reads.

Cost Implications: Early rejection saves downstream compute. Quarantine path enables investigation without blocking pipeline.