What are Assert Transformations, and where are they used?

Assert transformations validate data quality within a pipeline. They check conditions (e.g., no nulls, value ranges, referential integrity) and fail the pipeline if violated. Used in: dbt (schema and data tests), Great Expectations, custom Spark/Python checks. Example—dbt: tests: - unique: user_id - not_null: email. In Spark: df.filter("amount < 0").count() and assert == 0. Best practice: Place asserts after each critical transformation; use severity levels (warn vs fail); log context (row counts, sample failures) for debugging. Assert early in the pipeline to fail fast and avoid corrupting downstream tables. Why it matters: Design choices compound at scale—wrong approach can cause 100× overhead. Scalability trade-offs: Profile before optimizing; validate on sample then full. Cost implications: Suboptimal choices multiply at billion-row scale.