updates following martin feedback
This commit is contained in:
Allan Bowe
parent
554d58a5ef
commit
fbff27debe
@ -1,44 +1,21 @@
|
||||
# Data Controller for SAS: Overview
|
||||
# Data Controller for SAS®: Overview
|
||||
|
||||
## What does the Data Controller do?
|
||||
|
||||
The Data Controller for SAS allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity.
|
||||
The Data Controller allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity.
|
||||
|
||||
## What is a Target Table?
|
||||
A Target Table is a physical table, such as a SAS dataset or a Table in a database. The attributes of this table (eg Primary Key, loadtype, library, SCD variables etc) will have been predefined by your administrator so that you can change the data in that table safely and easily.
|
||||
|
||||
## Why do I need the Data Controller?
|
||||
|
||||
Alternatives to using the Data Controller for performing data updates include:
|
||||
|
||||
- Writing SAS or SQL code yourself to perform data updates
|
||||
- Asking your DBA to to perform updates (following a change mangement process)
|
||||
- Saving CSVs / data on a shared-visibility network drive and building / running jobs to perform uploads in batch
|
||||
- Opening (and therefore locking) datasets in Enterprise Guide or SAS® Table Viewer to perform direct updates
|
||||
- Building a custom web application yourself to perform secure updates
|
||||
|
||||
Problems with the above approaches include one or more of the following:
|
||||
|
||||
- Risk of manual error / data corruption
|
||||
- End users requiring direct write access to critical data sources in production
|
||||
- Breaches due to unnecessary parties having access to the data
|
||||
- Inability to trace who made the change, when, and why
|
||||
- Reliance on key individuals to perform updates
|
||||
- Requirement to build load routines every time a new data source is added
|
||||
- Upload routines have to be manually modified when the table changes
|
||||
- Legacy 'black box' solutions with little to no testing, documentation or support
|
||||
|
||||
The Data Controller for SAS® solves the issues above.
|
||||
|
||||
## How does it work?
|
||||
|
||||
From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (wich may also be the editor, depeneding on cconfiguration) reviews the changes and accepts / or rejects. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded.
|
||||
From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (which may also be the editor, depending on configuration) reviews the changes and accepts / or rejects them. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded.
|
||||
|
||||
## Who is it for?
|
||||
|
||||
There are 5 roles identified for users of the Data Controller:
|
||||
|
||||
1. *Viewer*. A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (share a url to share a table with a colleague, for instance).
|
||||
1. *Viewer*. A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (eg copy the url to share a table with a colleague).
|
||||
2. *Editor*. An editor makes changes to data in a table (add, modify, delete) and submits those changes to the approver(s) for acceptance.
|
||||
3. *Approver*. An approver accepts / rejects proposed changes to data under their control. If accepted, the change is applied to the target table.
|
||||
4. *Auditor*. An auditor has the ability to review the [history](dc-userguide.md#history) of changes to a particular table.
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
# Data Controller for SAS: User Guide
|
||||
# Data Controller for SAS: User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
@ -7,12 +7,12 @@ The Data Controller has 5 tabs, as follows:
|
||||
* *Viewer*. This tab lets users view any table to which they have been granted access in metadata. They can also download the data as csv, excel, or as a SAS program (datalines).
|
||||
* *Editor*. This tab enables users to add, modify or delete data. This can be done directly in the browser, or by uploading a CSV file. Values can also be copy-pasted from a spreadsheet. Once changes are ready, they can be submitted, with a corresponding reason.
|
||||
* *Submissions*. This shows and editor the outstanding changes that have been submitted for approval (but have not yet been approved or rejected).
|
||||
* *Approvals*. This shows an approver all the outstanding changes that are waiting to be approved.
|
||||
* *History*. This shows an auditor, or other interested party, what changes have been submitted for each table over time.
|
||||
* *Approvals*. This shows an approver all their outstanding approval requests.
|
||||
* *History*. This shows an auditor, or other interested party, what changes have been submitted for each table.
|
||||
|
||||
## Viewer
|
||||
|
||||
The Viewer screen lets any user with a SAS profile view tables to which they have already been granted access in metadata. Simply Select libary / table and the View button. The first 5,000 rows of the table in question are displayed.
|
||||
The Viewer screen lets any user with a SAS profile view tables to which they have already been granted access in metadata. Simply Select library / table and the View button. The first 5,000 rows of the table in question are displayed.
|
||||
|
||||
It is also possible to build complex filters against data before viewing, via the Filter button. The filter string is converted into an ID, as can be seen in the URL. Simply share this link with any other SAS user to share that particular view.
|
||||
The Viewer also has a Download option. This lets you Download your view of the data in CSV, Excel, and SAS format. The SAS format option gives you a SAS program with the relevant DATALINES so that you can easily recreate your data in another instance of SAS.
|
||||
@ -25,16 +25,14 @@ The Editor screen lets users who have been pre-authorised (via the `DATACTRL.MPE
|
||||
|
||||
2 - *Upload*. The user can upload a CSV file directly, instead of using the interface. The CSV must have the same structure as the target. Use the 'download csv' option in Viewer to obtain a template of this CSV.
|
||||
|
||||
3 - *Edit*. The main interface, explained below.
|
||||
|
||||
In the main interface, data is displayed in tabular format. The first column is always "Delete?". This allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
|
||||
3 - *Edit*. This is the main interface, data is displayed in tabular format. The first column is always "Delete?", as this allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
|
||||
The next set of columns are the Primary Key, and are shaded grey. If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend). For SCD2 type tables, the 'validity' fields are not shown. It is assumed that the user is always working with the current version of the data, and the view is filtered as such.
|
||||
After this, remaining columns are shown. Dates / datetime fields have appropriate datepickers. Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the `DATACTRL.MPE_SELECTBOX` table.
|
||||
|
||||
New rows can be added using the right click context menu, or the 'Add Row' button. The data can also be sorted by clicking on the column headers.
|
||||
|
||||
When ready to submit, hit the SUBMIT button and enter a reason for the change. The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen.
|
||||
If you are also an approver you can approve this change yourself (and you have a button now available so you can approve immediately). See Approve section.
|
||||
If you are also an approver you can approve this change yourself.
|
||||
|
||||
## Submitted
|
||||
This page shows a list of the changes you have submitted (that are not yet approved).
|
||||
@ -43,4 +41,4 @@ This page shows a list of the changes you have submitted (that are not yet appro
|
||||
This shows the list of changes that have been submitted to you (or your group) for approval.
|
||||
|
||||
## History
|
||||
View the list of changes to each table, who made the change, when, etc.
|
||||
View the list of changes to each table, who made the change, when, etc.
|
||||
|
||||
@ -19,7 +19,7 @@ The libref of the table. If not pre-assigned, DC will assign it at runtime usin
|
||||
The dataset (table) name as visible when assigning a direct libref connection to `LIBREF`.
|
||||
|
||||
### NUM_OF_APPROVALS_REQUIRED
|
||||
This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, say, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission.
|
||||
This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, for example, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission.
|
||||
This is a required field.
|
||||
|
||||
### LOADTYPE
|
||||
@ -50,11 +50,11 @@ Set the name of a variable (eg `processed_dttm`) which should be given a current
|
||||
Leave blank if not required.
|
||||
|
||||
### CLOSE_VARS
|
||||
By default, the data controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the _absence_ of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The `CLOSE_VARS` are typically a subset of the [BUSKEY](#buskey) fields.
|
||||
By default, the Data Controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the _absence_ of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The `CLOSE_VARS` are typically a subset of the [BUSKEY](#buskey) fields.
|
||||
Leave blank if not required.
|
||||
|
||||
### PRE_EDIT_HOOK
|
||||
The full path / location (unquoted) of a SAS program that will be `%inc`'d prior to an edit being made. This allows a particular view of a table to be presented to a user for editing (eg masking columns etc).
|
||||
The full path / location (unquoted) of a SAS program that will be `%inc`'d prior to an edit being made. This allows a particular view of a table to be presented to a user for editing (eg masking columns etc).
|
||||
Leave blank if not required.
|
||||
|
||||
### POST_EDIT_HOOK
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
#Data Controller for SAS® - Backend Deployment
|
||||
|
||||
## Overview
|
||||
The backend for data controller consists of a set of Stored Processes, a macro library, and a database. The database can be SAS Base library if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
|
||||
The backend for Data Controller consists of a set of Stored Processes, a macro library, and a database. The database can be SAS Base library if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
|
||||
|
||||
## Details
|
||||
|
||||
@ -9,11 +9,11 @@ The backend for data controller consists of a set of Stored Processes, a macro l
|
||||
|
||||
2 - Import datacontroller.spk using SAS Management Console. The location in which this is deployed should be added to the `metadataRoot` value in the `h54s.config` file as per [frontend](dci-frontend.md#details) deployment.
|
||||
|
||||
3 - Create a staging directory. This will contain the data submitted by editors and awaiting approval. The Stored Process system account (eg `sassrv`) will need write access to this location.
|
||||
3 - Create a staging directory. This will contain the data submitted by editors and awaiting approval. The Stored Process system account (eg `sassrv`) will need write access to this location.
|
||||
|
||||
4 - Register a library in metadata for your preferred database. The libref should be `DATACTRL`. If this is not possible, then an alternative libref can be used, simply specify it in the configuration in step 6.
|
||||
|
||||
5 - Make the `mpeinit.sas` macro available. This "initialisation" macro needs to be available to a fresh Stored Process session. There are several ways to do this, such as to `%include` from the `/Lev1/StoredProcessServer/StoredProcessServer_usermods.sas` file or to create a symlink from the `sasautos` directory.
|
||||
5 - Make the `mpeinit.sas` macro available. This "initialisation" macro needs to be available to a fresh Stored Process session. There are several ways to do this, such as to `%include` from the `/Lev1/StoredProcessServer/StoredProcessServer_usermods.sas` file or to create a symlink from the `sasautos` directory.
|
||||
|
||||
6 - Configure the `mpeinit.sas` file. This is where all your site specific locations are stored. The following macro variables should be modified:
|
||||
|
||||
|
||||
@ -5,11 +5,11 @@ The Data Controller front end was built on the Angular seed app from Boemska App
|
||||
|
||||
## Instructions
|
||||
|
||||
1 - Unzip dcfrontend.zip and upload the entire `datacontroller` directory to the static content server
|
||||
1 - Unzip dcfrontend.zip and upload the entire `datacontroller` directory to the static content server.
|
||||
|
||||
2 - Open the `h54s.config` file and update the `metadataRoot` value to the location of the Stored Processes as per [backend](dci-backend.md) deployment.
|
||||
|
||||
It should now be possible to use the application - simpliy navigate to YO
|
||||
URSASWEBLOC.domain/yourRoot/datacontroller and sign in!
|
||||
It should now be possible to use the application - simply navigate to `YO
|
||||
URSASWEBLOC.domain/yourRoot/datacontroller` and sign in!
|
||||
|
||||
The next step is to [configure](dcc-tables.md) the tables.
|
||||
|
||||
@ -2,13 +2,13 @@
|
||||
|
||||
## Overview
|
||||
|
||||
The Data Controller for SAS® enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving each update.
|
||||
The Data Controller for SAS® enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving them.
|
||||
|
||||
<iframe src="https://player.vimeo.com/video/277472582" width="640" height="360" frameborder="0" allowfullscreen></iframe>
|
||||
|
||||
## Documents
|
||||
|
||||
The following documents are available on this site:
|
||||
The following information is available on this site:
|
||||
|
||||
* [Overview](dc-overview.md)
|
||||
* [User Guide](dc-userguide.md)
|
||||
@ -17,7 +17,7 @@ The following documents are available on this site:
|
||||
|
||||
## Online resources
|
||||
|
||||
The following sites contain additional information on the Data Controller for SAS®:
|
||||
The following sites contain additional information on the Data Controller:
|
||||
|
||||
* DC Overview [Presentation](https://slides.com/allanbowe/datacontroller)
|
||||
* Data Controller [LinkedIn](https://www.linkedin.com/company/datacontroller/) page
|
||||
|
||||
Loading…
Reference in New Issue
Block a user