Restoring Historical Data Versions #84

New Issue

allan · 2024-03-07T13:26:24Z

allan commented

2024-03-07 13:26:24 +00:00

There are three possibilities for restoring historical versions in DC:

Restoring based on a timestamp (relevant for TXTEMPORAL / SCD2 loads)
- Only relevant for SCD2 loads
Restoring based on the information in the MPE_AUDIT table
- Only possible where the table has an audit history
Restoring based on the diffs.csv file in the staging location
- Always possible but more complicated as multiple locations to source from

There are many complications with a restore process, such as:

Dealing with model changes
- Need to drop or nullify data as appropriate
Dealing with changes not made by Data Controller
- Unable to address this
Dealing with complex loads
- Bitemporal restores are out of scope for now

In building a restore process, a user interface will be presented showing historical versions, driven from the MPE_DATALOADS table. The user will be able to choose a previous version, Data Controller will compare the current and previous versions, and a new staged load will be submitted (for approval) as the final step in the process.

A "bonus" step in delivering the "restore historical version" feature will also be the ability to "view historical version".

There are three possibilities for restoring historical versions in DC: * Restoring based on a timestamp (relevant for TXTEMPORAL / SCD2 loads) - Only relevant for SCD2 loads * Restoring based on the information in the MPE_AUDIT table - Only possible where the table has an audit history * Restoring based on the diffs.csv file in the staging location - Always possible but more complicated as multiple locations to source from There are many complications with a restore process, such as: * Dealing with model changes - Need to drop or nullify data as appropriate * Dealing with changes not made by Data Controller - Unable to address this * Dealing with complex loads - Bitemporal restores are out of scope for now In building a restore process, a user interface will be presented showing historical versions, driven from the MPE_DATALOADS table. The user will be able to choose a previous version, Data Controller will compare the current and previous versions, and a new staged load will be submitted (for approval) as the final step in the process. A "bonus" step in delivering the "restore historical version" feature will also be the ability to "view historical version".

allan referenced this issue from a commit

2024-04-29 23:20:29 +00:00

feat: SAS services & tests for RESTORE, #84

allan referenced a pull request that will close this issue

2024-04-29 23:27:24 +00:00

Restore Previous State / Data Rollback #94

allan referenced this issue from a commit

2024-04-30 16:45:50 +00:00

fix: issue with multiple adds/deletes, #84

allan closed this issue

2024-05-02 11:18:29 +00:00

gitea-actions referenced this issue from a commit