dc/docs.datacontroller.io
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updates following martin feedback

Browse Source
This commit is contained in:
Allan Bowe 2018-07-09 16:38:11 +02:00
parent 554d58a5ef
commit fbff27debe
6 changed files with 23 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
docs/dc-overview.md
View File

@ -1,44 +1,21 @@
# Data Controller for SAS: Overview
# Data Controller for SAS®: Overview
## What does the Data Controller do?
The Data Controller for SAS allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity.
The Data Controller allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity.
## What is a Target Table?
A Target Table is a physical table, such as a SAS dataset or a Table in a database. The attributes of this table (eg Primary Key, loadtype, library, SCD variables etc) will have been predefined by your administrator so that you can change the data in that table safely and easily.
## Why do I need the Data Controller?
Alternatives to using the Data Controller for performing data updates include:
- Writing SAS or SQL code yourself to perform data updates
- Asking your DBA to to perform updates (following a change mangement process)
- Saving CSVs / data on a shared-visibility network drive and building / running jobs to perform uploads in batch
- Opening (and therefore locking) datasets in Enterprise Guide or SAS® Table Viewer to perform direct updates
- Building a custom web application yourself to perform secure updates
Problems with the above approaches include one or more of the following:
- Risk of manual error / data corruption
- End users requiring direct write access to critical data sources in production
- Breaches due to unnecessary parties having access to the data
- Inability to trace who made the change, when, and why
- Reliance on key individuals to perform updates
- Requirement to build load routines every time a new data source is added
- Upload routines have to be manually modified when the table changes
- Legacy 'black box' solutions with little to no testing, documentation or support
The Data Controller for SAS® solves the issues above.
## How does it work?
From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (wich may also be the editor, depeneding on cconfiguration) reviews the changes and accepts / or rejects. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded.
From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (which may also be the editor, depending on configuration) reviews the changes and accepts / or rejects them. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded.
## Who is it for?
There are 5 roles identified for users of the Data Controller:
1. *Viewer*. A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (share a url to share a table with a colleague, for instance).
1. *Viewer*. A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (eg copy the url to share a table with a colleague).
2. *Editor*. An editor makes changes to data in a table (add, modify, delete) and submits those changes to the approver(s) for acceptance.
3. *Approver*. An approver accepts / rejects proposed changes to data under their control. If accepted, the change is applied to the target table.
4. *Auditor*. An auditor has the ability to review the [history](dc-userguide.md#history) of changes to a particular table.

12
docs/dc-userguide.md
View File

@ -7,12 +7,12 @@ The Data Controller has 5 tabs, as follows:
* *Viewer*. This tab lets users view any table to which they have been granted access in metadata. They can also download the data as csv, excel, or as a SAS program (datalines).
* *Editor*. This tab enables users to add, modify or delete data. This can be done directly in the browser, or by uploading a CSV file. Values can also be copy-pasted from a spreadsheet. Once changes are ready, they can be submitted, with a corresponding reason.
* *Submissions*. This shows and editor the outstanding changes that have been submitted for approval (but have not yet been approved or rejected).
* *Approvals*. This shows an approver all the outstanding changes that are waiting to be approved.
* *History*. This shows an auditor, or other interested party, what changes have been submitted for each table over time.
* *Approvals*. This shows an approver all their outstanding approval requests.
* *History*. This shows an auditor, or other interested party, what changes have been submitted for each table.
## Viewer
The Viewer screen lets any user with a SAS profile view tables to which they have already been granted access in metadata. Simply Select libary / table and the View button. The first 5,000 rows of the table in question are displayed.
The Viewer screen lets any user with a SAS profile view tables to which they have already been granted access in metadata. Simply Select library / table and the View button. The first 5,000 rows of the table in question are displayed.
It is also possible to build complex filters against data before viewing, via the Filter button. The filter string is converted into an ID, as can be seen in the URL. Simply share this link with any other SAS user to share that particular view.
The Viewer also has a Download option. This lets you Download your view of the data in CSV, Excel, and SAS format. The SAS format option gives you a SAS program with the relevant DATALINES so that you can easily recreate your data in another instance of SAS.
@ -25,16 +25,14 @@ The Editor screen lets users who have been pre-authorised (via the `DATACTRL.MPE
2 - *Upload*. The user can upload a CSV file directly, instead of using the interface. The CSV must have the same structure as the target. Use the 'download csv' option in Viewer to obtain a template of this CSV.
3 - *Edit*. The main interface, explained below.
In the main interface, data is displayed in tabular format. The first column is always "Delete?". This allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
3 - *Edit*. This is the main interface, data is displayed in tabular format. The first column is always "Delete?", as this allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted.
The next set of columns are the Primary Key, and are shaded grey. If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend). For SCD2 type tables, the 'validity' fields are not shown. It is assumed that the user is always working with the current version of the data, and the view is filtered as such.
After this, remaining columns are shown. Dates / datetime fields have appropriate datepickers. Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the `DATACTRL.MPE_SELECTBOX` table.
New rows can be added using the right click context menu, or the 'Add Row' button. The data can also be sorted by clicking on the column headers.
When ready to submit, hit the SUBMIT button and enter a reason for the change. The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen.
If you are also an approver you can approve this change yourself (and you have a button now available so you can approve immediately). See Approve section.
If you are also an approver you can approve this change yourself.
## Submitted
This page shows a list of the changes you have submitted (that are not yet approved).

4
docs/dcc-tables.md
View File

@ -19,7 +19,7 @@ The libref of the table. If not pre-assigned, DC will assign it at runtime usin
The dataset (table) name as visible when assigning a direct libref connection to `LIBREF`.
### NUM_OF_APPROVALS_REQUIRED
This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, say, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission.
This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, for example, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission.
This is a required field.
### LOADTYPE
@ -50,7 +50,7 @@ Set the name of a variable (eg `processed_dttm`) which should be given a current
Leave blank if not required.
### CLOSE_VARS
By default, the data controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the _absence_ of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The `CLOSE_VARS` are typically a subset of the [BUSKEY](#buskey) fields.
By default, the Data Controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the _absence_ of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The `CLOSE_VARS` are typically a subset of the [BUSKEY](#buskey) fields.
Leave blank if not required.
### PRE_EDIT_HOOK

2
docs/dci-backend.md
View File

@ -1,7 +1,7 @@
#Data Controller for SAS® - Backend Deployment
## Overview
The backend for data controller consists of a set of Stored Processes, a macro library, and a database. The database can be SAS Base library if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
The backend for Data Controller consists of a set of Stored Processes, a macro library, and a database. The database can be SAS Base library if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
## Details

6
docs/dci-frontend.md
View File

@ -5,11 +5,11 @@ The Data Controller front end was built on the Angular seed app from Boemska App
## Instructions
1 - Unzip dcfrontend.zip and upload the entire `datacontroller` directory to the static content server
1 - Unzip dcfrontend.zip and upload the entire `datacontroller` directory to the static content server.
2 - Open the `h54s.config` file and update the `metadataRoot` value to the location of the Stored Processes as per [backend](dci-backend.md) deployment.
It should now be possible to use the application - simpliy navigate to YO
URSASWEBLOC.domain/yourRoot/datacontroller and sign in!
It should now be possible to use the application - simply navigate to `YO
URSASWEBLOC.domain/yourRoot/datacontroller` and sign in!
The next step is to [configure](dcc-tables.md) the tables.

6
docs/index.md
View File

@ -2,13 +2,13 @@
## Overview
The Data Controller for SAS® enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving each update.
The Data Controller for SAS® enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving them.
<iframe src="https://player.vimeo.com/video/277472582" width="640" height="360" frameborder="0" allowfullscreen></iframe>
## Documents
The following documents are available on this site:
The following information is available on this site:
* [Overview](dc-overview.md)
* [User Guide](dc-userguide.md)
@ -17,7 +17,7 @@ The following documents are available on this site:
## Online resources
The following sites contain additional information on the Data Controller for SAS®:
The following sites contain additional information on the Data Controller:
* DC Overview [Presentation](https://slides.com/allanbowe/datacontroller)
* Data Controller [LinkedIn](https://www.linkedin.com/company/datacontroller/) page