Explain Delta Live Tables and their features, such as declarative pipeline definition and automatic data validation.

Why it matters: At scale, design choices directly impact reliability, latency, and cost. Wrong decisions compound across jobs and teams.

Delta Live Tables (DLT) is Databricks' declarative framework for building production data pipelines. It enables declarative pipeline definition through Python or SQL, where you define what the data should look like rather than imperative transformation steps. Key features include: (1) Declarative pipeline definition—tables are defined with EXPECT, ASSERT, and constraints; (2) Automatic data validation via expectations that fail, drop, or quarantine bad records; (3) Built-in monitoring and lineage; (4) Automatic scaling and cluster management. Example: @dlt.table(name='bronze_sales') def bronze_sales(): return spark.readStream.format('delta').load('/path').expect('valid_amount', 'amount >= 0').expectOrFail('valid_id', 'id IS NOT NULL'). In production, use expectation rules for SLAs, enable auto-optimize and auto-compaction for Delta tables, and leverage DLT's built-in retry and error handling.

Scalability trade-offs: Partition/parallelism limits; single points of failure; horizontal vs vertical scaling. Cost implications: Sizing, spot vs reserved, optimization ROI.