How do you see files before update (history records/versioning)?

To see files before an update, data engineers primarily leverage version control systems for code and configuration, object storage versioning for raw data files, and data lake table formats for structured data tables.

Mechanics and Why

Version Control Systems (VCS) like Git track changes to code, scripts, and configuration files. Each commit creates a snapshot of the repository, allowing you to view, compare, or revert to any previous state using commands like git log to see commit history, git show <commit_hash>:<filepath> to inspect a file at a specific commit, or git diff to compare versions. This is essential for managing development lifecycles and auditing changes to data pipelines themselves.

Object Storage Versioning (e.g., AWS S3 Versioning) automatically retains multiple versions of an object whenever it's modified or deleted. When enabled on a bucket, every PUT operation creates a new version, and DELETE operations add a delete marker, preserving previous object states. This is critical for recovering from accidental deletions or overwrites of raw data files, providing a simple history at the object level without needing application-level logic.

Data Lake Table Formats (e.g., Delta Lake, Apache Iceberg) offer robust 'time travel' capabilities for structured and semi-structured data. These formats maintain a transaction log (or commit log) that records every operation (inserts, updates, deletes, schema changes). By consulting this log, the system can reconstruct the table's state at any past point in time, allowing queries against historical versions of the data. This provides a powerful mechanism for auditing, reproducing results, or rolling back data errors directly within the data table.

Concrete Example and Trade-offs

For example, with Delta Lake, you can query a table as it existed at a specific version or timestamp:

SELECT * FROM my_delta_table VERSION AS OF 123;
-- Or: SELECT * FROM my_delta_table TIMESTAMP AS OF '2023-10-26 10:00:00';

You can also use DESCRIBE HISTORY my_delta_table to view the transaction log. While versioning and historical data retention add storage overhead, the benefits of data integrity, auditability, and disaster recovery typically outweigh the costs, especially for critical data assets.

In the interview, also mention…

In the interview, also mention database backups, Change Data Capture (CDC) systems, and dbt snapshots as complementary approaches for historical data.