When would you architecturally choose Dataset[T] over DataFrame in a Scala Spark pipeline, and what are the scalability and portability trade-offs? Include type-safety benefits vs. operational constraints.

DataFrame is an untyped collection of Row objects with schema at runtime; Dataset[T] is typed with compile-time safety. In Scala, DataFrame = Dataset[Row]. Architectural why: Dataset enables domain modeling (e.g., Dataset[Order])—catch errors at compile time, better IDE support, and Catalyst can optimize typed encoders. Scalability: both use Tungsten and Catalyst; Dataset adds encoder overhead but marginal for most workloads. Portability trade-off: PySpark has only DataFrame—no typed Dataset. Choosing Dataset in Scala ties the codebase to Scala; DataFrame is cross-language. Cost implication: Dataset reduces runtime bugs (fewer prod incidents) but increases maintenance if the team shifts to Python. Use Dataset when: Scala-heavy team, domain-rich ETL, compile-time guarantees matter. Use DataFrame when: multi-language org, SQL-first, quick iteration. Production: prefer DataFrame for portability; Dataset when team is Scala-centric and wants stronger typing.