{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":true,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Data Controller Documentation \u00b6 Overview \u00b6 The Data Controller for SAS\u00ae enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving them. Online resources \u00b6 The following resources contain additional information on the Data Controller: DC Overview Presentation Data Controller LinkedIn page Data Controller website Data Controller flyer ( front / back ) Data Controller videos Data Controller SAS Code Data Controller Download Product Features \u00b6 Data Controller is regularly updated with new features. If you see something that is not listed, and we agree it would be useful, you can engage us with Developer Days to build the feature in. Excel uploads - drag & drop directly into SAS. All versions of excel supported. Data Lineage - at both table and column level, export as image or CSV Data Validation Rules - both automatic and user defined Data Dictionary - map data definitions and ownership Data Catalog - including primary key extraction DDL generator - in SAS, TSQL and PGSQL flavours Workflow - run a SAS program before or after an edit or approval Role based security - editors, approvers, administrators Data Models supported: SCD2, Bitemporal, composite keys, retained keys, update and full replace Email Alerts for data changes Row Level Security Excel formula support Dynamic cell dropdown Works on ALL flavours of SAS (Foundation, EBI, Viya)","title":"Home"},{"location":"#data-controller-documentation","text":"","title":"Data Controller Documentation"},{"location":"#overview","text":"The Data Controller for SAS\u00ae enables users to self serve their data changes, and for data owners to retain control over those updates by reviewing and approving them.","title":"Overview"},{"location":"#online-resources","text":"The following resources contain additional information on the Data Controller: DC Overview Presentation Data Controller LinkedIn page Data Controller website Data Controller flyer ( front / back ) Data Controller videos Data Controller SAS Code Data Controller Download","title":"Online resources"},{"location":"#product-features","text":"Data Controller is regularly updated with new features. If you see something that is not listed, and we agree it would be useful, you can engage us with Developer Days to build the feature in. Excel uploads - drag & drop directly into SAS. All versions of excel supported. Data Lineage - at both table and column level, export as image or CSV Data Validation Rules - both automatic and user defined Data Dictionary - map data definitions and ownership Data Catalog - including primary key extraction DDL generator - in SAS, TSQL and PGSQL flavours Workflow - run a SAS program before or after an edit or approval Role based security - editors, approvers, administrators Data Models supported: SCD2, Bitemporal, composite keys, retained keys, update and full replace Email Alerts for data changes Row Level Security Excel formula support Dynamic cell dropdown Works on ALL flavours of SAS (Foundation, EBI, Viya)","title":"Product Features"},{"location":"admin-services/","text":"Admin Services \u00b6 Several web services have been defined to provide additional functionality outside of the user interface. These somewhat-hidden services must be called directly, using a web browser. In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs). The URL is made up of several components: SERVERURL -> the domain (and port) on which your SAS server resides EXECUTOR -> Either SASStoredProcess for SAS 9, else SASJobExecution for Viya APPLOC -> The root folder location in which the Data Controller backend services were deployed SERVICE -> The actual Data Controller service being described. May include additional parameters. To illustrate the above, consider the following URL: https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL This is broken down into: $SERVERURL = https://sas.analytium.co.uk $EXECUTOR = SASJobExecution $APPLOC = /Public/app/dc $SERVICE = services/admin/exportdb&flavour=PGSQL The below sections will only describe the $SERVICE component - you may construct this into a URL as follows: $SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE Export Config \u00b6 This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance. EXAMPLE: services/admin/exportconfig Export Database \u00b6 Exports the data controller control library in DB specific DDL. The following URL parameters may be added: &flavour= (only PGSQL supported at this time) &schema= (optional, if target schema is needed) EXAMPLES: services/admin/exportdb&flavour=PGSQL&schema=DC services/admin/exportdb&flavour=PGSQL Refresh Data Catalog \u00b6 In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as: Library Properties (size, schema, path, number of tables) Table Properties (size, number of columns, primary keys) Variable Properties (presence in a primary key, constraints, position in the dataset) The data is stored with SCD2 so you can actually track changes to your model over time ! Curious when that new column appeared? Just check the history in MPE_DATACATALOG_TABS . To run the refresh process, just trigger the stored process, eg below: services/admin/refreshcatalog services/admin/refreshcatalog&libref=MYLIB The optional &libref= parameter allows you to run the process for a single library. Just provide the libref. When doing a full scan, the following LIBREFS are ignored: 'CASUSER' 'MAPSGFK' 'SASUSER' 'SASWORK 'STPSAMP' 'TEMP' `WORK' Additional LIBREFs can be excluded by adding them to the DCXXXX.MPE_CONFIG table (where var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS' ). Use a pipe ( | ) symbol to seperate them. This can be useful where there are connection issues for a particular library. Be aware that the scan process can take a long time if you have a lot of tables! Output tables (all SCD2): MPE_DATACATALOG_LIBS - Library attributes MPE_DATACATALOG_TABS - Table attributes MPE_DATACATALOG_VARS - Column attributes MPE_DATASTATUS_LIBS - Frequently changing library attributes (such as size & number of tables) MPE_DATASTATUS_TABS - Frequently changing table attributes (such as size & number of rows) Update Licence Key \u00b6 Whenever navigating Data Controller, there is always a hash ( # ) in the URL. To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: /licensing/update . If you are using https protocol, you will have 2 keys (licence key / activation key). In http mode, there is just one key (licence key) for both boxes.","title":"Admin Services"},{"location":"admin-services/#admin-services","text":"Several web services have been defined to provide additional functionality outside of the user interface. These somewhat-hidden services must be called directly, using a web browser. In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs). The URL is made up of several components: SERVERURL -> the domain (and port) on which your SAS server resides EXECUTOR -> Either SASStoredProcess for SAS 9, else SASJobExecution for Viya APPLOC -> The root folder location in which the Data Controller backend services were deployed SERVICE -> The actual Data Controller service being described. May include additional parameters. To illustrate the above, consider the following URL: https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL This is broken down into: $SERVERURL = https://sas.analytium.co.uk $EXECUTOR = SASJobExecution $APPLOC = /Public/app/dc $SERVICE = services/admin/exportdb&flavour=PGSQL The below sections will only describe the $SERVICE component - you may construct this into a URL as follows: $SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE","title":"Admin Services"},{"location":"admin-services/#export-config","text":"This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance. EXAMPLE: services/admin/exportconfig","title":"Export Config"},{"location":"admin-services/#export-database","text":"Exports the data controller control library in DB specific DDL. The following URL parameters may be added: &flavour= (only PGSQL supported at this time) &schema= (optional, if target schema is needed) EXAMPLES: services/admin/exportdb&flavour=PGSQL&schema=DC services/admin/exportdb&flavour=PGSQL","title":"Export Database"},{"location":"admin-services/#refresh-data-catalog","text":"In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as: Library Properties (size, schema, path, number of tables) Table Properties (size, number of columns, primary keys) Variable Properties (presence in a primary key, constraints, position in the dataset) The data is stored with SCD2 so you can actually track changes to your model over time ! Curious when that new column appeared? Just check the history in MPE_DATACATALOG_TABS . To run the refresh process, just trigger the stored process, eg below: services/admin/refreshcatalog services/admin/refreshcatalog&libref=MYLIB The optional &libref= parameter allows you to run the process for a single library. Just provide the libref. When doing a full scan, the following LIBREFS are ignored: 'CASUSER' 'MAPSGFK' 'SASUSER' 'SASWORK 'STPSAMP' 'TEMP' `WORK' Additional LIBREFs can be excluded by adding them to the DCXXXX.MPE_CONFIG table (where var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS' ). Use a pipe ( | ) symbol to seperate them. This can be useful where there are connection issues for a particular library. Be aware that the scan process can take a long time if you have a lot of tables! Output tables (all SCD2): MPE_DATACATALOG_LIBS - Library attributes MPE_DATACATALOG_TABS - Table attributes MPE_DATACATALOG_VARS - Column attributes MPE_DATASTATUS_LIBS - Frequently changing library attributes (such as size & number of tables) MPE_DATASTATUS_TABS - Frequently changing table attributes (such as size & number of rows)","title":"Refresh Data Catalog"},{"location":"admin-services/#update-licence-key","text":"Whenever navigating Data Controller, there is always a hash ( # ) in the URL. To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: /licensing/update . If you are using https protocol, you will have 2 keys (licence key / activation key). In http mode, there is just one key (licence key) for both boxes.","title":"Update Licence Key"},{"location":"api/","text":"Warning Work in Progress! API \u00b6 Where a project has a requirement to load Excel Files automatically into SAS, from a remote machine, an API approach is desirable for many reasons: Security. Client access can be limited to just the endpoints they need (rather than being granted full server access). Flexibility. Well documented, stable APIs allow consumers to build and extend additional products and solutions. Cost. API solutions are typically self-contained, quick to implement, and easy to learn. A Data Controller API would enable teams across an entire enterprise to easily and securely send data to SAS in a transparent and fully automated fashion. The API would also benefit from all of Data Controllers existing data validation logic (both frontend and backend), data auditing, alerts , and security control features. It is however, a significant departure from the existing \"SAS Content\" based deployment, in the following ways: Server Driven. A machine is required on which to launch, and run, the API application itself. Fully Automated. There is no browser, or interface, or - human, involved. Extends outside of SAS. There are firewalls, and authentication methods, to consider. The Data Controller technical solution will differ, depending on the type of SAS Platform being used. There are three types of SAS Platform: Foundation SAS - regular, Base SAS. SAS EBI - with Metadata. SAS Viya - cloud enabled. And there are three main options when it comes to building APIs on SAS: Standalone DC API (Viya Only). Viya comes with REST APIs out of the box, no middleware needed. SAS 9 API . This is an open-source Java Application, using SAS Authentication. SASjs Server . An open source NodeJS application, compatible with all major authentication methods and all versions of SAS An additional REST API option for SAS EBI might have been BI Web Services , however - it requires platform changes and is not highly secure. The compatibility matrix is therefore as follows: Product Foundation SAS SAS EBI SAS VIYA DCAPI \u274c \u274c \u2705 DCAPI + SASjs Server \u2705 \u2705 \u2705 DCAPI + SAS 9 API \u274c \u2705 \u274c In all cases, a Data Controller API will be surfaced, that makes use of the underlying (raw) API server. The following sections break down these options, and the work remaining to make them a reality. Standalone DC API (Viya Only) \u00b6 For Viya, the investment necessary is relatively low, thanks to the API-first nature of the platform. In addition, the SASjs framework already provides most of the necessary functionality - such as authentication, service execution, handling results & logs, etc. Finally, the Data Controller team have already built an API Bridge (specific to another customer, hence the building blocks are in place). The work to complete the Viya version of the API is as follows: Authorisation interface Creation of API services Tests & Automated Deployments Developer docs Swagger API Public Documentation Cost to complete - \u00a35,000 (Viya Only) SASjs Server (Foundation SAS) \u00b6 SASjs Server already provides an API interface over Foundation SAS. An example of building a web app using SASjs Server can be found here . In order for it to fulfill the role as the engine behind the Data Controller API, additional work is needed - specifically: Secure (Enterprise) Authentication Users & Groups Feature configuration (ability to restrict features to different groups) On top of this, the DC API part would cover: Authorisation interface Creation of API services Tests & Automated Deployments Developer docs Swagger API Public Documentation Cost to complete - \u00a310,000 (fixed) Given that all three SAS platforms have Foundation SAS available, this option will work everywhere. The only restriction is that the sasjs/server instance must be located on the same server as SAS. ` SAS 9 API (SAS EBI) \u00b6 This product has one major benefit - there is nothing to install on the SAS Platform itself. It connects to SAS in much the same way as Enterprise Guide (using the SAS IOM). Website: https://sas9api.io Github: https://github.com/analytium/sas9api The downside is that the features needed by Data Controller are not present in the API. Furthermore, the tool is not under active development. To build out the necessary functionality, it will require us to source a senior Java developer on a short term contract to first, understand the tool, and secondly, to update it in a sustainable way. We estimate the cost to build Data Controller API on this mechanism at \u00a320,000 - but it could be higher.","title":"DC API"},{"location":"api/#api","text":"Where a project has a requirement to load Excel Files automatically into SAS, from a remote machine, an API approach is desirable for many reasons: Security. Client access can be limited to just the endpoints they need (rather than being granted full server access). Flexibility. Well documented, stable APIs allow consumers to build and extend additional products and solutions. Cost. API solutions are typically self-contained, quick to implement, and easy to learn. A Data Controller API would enable teams across an entire enterprise to easily and securely send data to SAS in a transparent and fully automated fashion. The API would also benefit from all of Data Controllers existing data validation logic (both frontend and backend), data auditing, alerts , and security control features. It is however, a significant departure from the existing \"SAS Content\" based deployment, in the following ways: Server Driven. A machine is required on which to launch, and run, the API application itself. Fully Automated. There is no browser, or interface, or - human, involved. Extends outside of SAS. There are firewalls, and authentication methods, to consider. The Data Controller technical solution will differ, depending on the type of SAS Platform being used. There are three types of SAS Platform: Foundation SAS - regular, Base SAS. SAS EBI - with Metadata. SAS Viya - cloud enabled. And there are three main options when it comes to building APIs on SAS: Standalone DC API (Viya Only). Viya comes with REST APIs out of the box, no middleware needed. SAS 9 API . This is an open-source Java Application, using SAS Authentication. SASjs Server . An open source NodeJS application, compatible with all major authentication methods and all versions of SAS An additional REST API option for SAS EBI might have been BI Web Services , however - it requires platform changes and is not highly secure. The compatibility matrix is therefore as follows: Product Foundation SAS SAS EBI SAS VIYA DCAPI \u274c \u274c \u2705 DCAPI + SASjs Server \u2705 \u2705 \u2705 DCAPI + SAS 9 API \u274c \u2705 \u274c In all cases, a Data Controller API will be surfaced, that makes use of the underlying (raw) API server. The following sections break down these options, and the work remaining to make them a reality.","title":"API"},{"location":"api/#standalone-dc-api-viya-only","text":"For Viya, the investment necessary is relatively low, thanks to the API-first nature of the platform. In addition, the SASjs framework already provides most of the necessary functionality - such as authentication, service execution, handling results & logs, etc. Finally, the Data Controller team have already built an API Bridge (specific to another customer, hence the building blocks are in place). The work to complete the Viya version of the API is as follows: Authorisation interface Creation of API services Tests & Automated Deployments Developer docs Swagger API Public Documentation Cost to complete - \u00a35,000 (Viya Only)","title":"Standalone DC API (Viya Only)"},{"location":"api/#sasjs-server-foundation-sas","text":"SASjs Server already provides an API interface over Foundation SAS. An example of building a web app using SASjs Server can be found here . In order for it to fulfill the role as the engine behind the Data Controller API, additional work is needed - specifically: Secure (Enterprise) Authentication Users & Groups Feature configuration (ability to restrict features to different groups) On top of this, the DC API part would cover: Authorisation interface Creation of API services Tests & Automated Deployments Developer docs Swagger API Public Documentation Cost to complete - \u00a310,000 (fixed) Given that all three SAS platforms have Foundation SAS available, this option will work everywhere. The only restriction is that the sasjs/server instance must be located on the same server as SAS. `","title":"SASjs Server (Foundation SAS)"},{"location":"api/#sas-9-api-sas-ebi","text":"This product has one major benefit - there is nothing to install on the SAS Platform itself. It connects to SAS in much the same way as Enterprise Guide (using the SAS IOM). Website: https://sas9api.io Github: https://github.com/analytium/sas9api The downside is that the features needed by Data Controller are not present in the API. Furthermore, the tool is not under active development. To build out the necessary functionality, it will require us to source a senior Java developer on a short term contract to first, understand the tool, and secondly, to update it in a sustainable way. We estimate the cost to build Data Controller API on this mechanism at \u00a320,000 - but it could be higher.","title":"SAS 9 API (SAS EBI)"},{"location":"column-level-security/","text":"Column Level Security \u00b6 Column level security is implemented by mapping allowed columns to a list of SAS groups. In VIEW mode, only allowed columns are visible. In EDIT mode, allowed columns are editable - the remaining columns are read-only. Below is an example of an EDIT table with only one column enabled for editing: See also: Row Level Security . Configuration \u00b6 The variables in MPE_COLUMN_LEVEL_SECURITY should be configured as follows: CLS_SCOPE \u00b6 Determines whether the rule applies to the VIEW page, the EDIT page, or ALL pages. The impact of the rule varies as follows: VIEW Scope \u00b6 When CLS_SCOPE in ('VIEW','ALL') then only the listed columns are visible (unless CLS_HIDE=1 ) EDIT Scope \u00b6 When CLS_SCOPE in ('EDIT','ALL') then only the listed columns are editable (the remaining columns are read-only, and visible). Furthermore: The user will be unable to ADD or DELETE records. Primary Key values are always read only Primary Key values cannot be hidden ( CLS_HIDE=1 will have no effect) CLS_GROUP \u00b6 The SAS Group to which the rule applies. The user could also be a member of a DC group . If a user is in ANY of the groups, the columns will be restricted. If a user is in NONE of the groups, no restrictions apply (all columns available). If a user is in MULTIPLE groups, they will see all allowed columns across all groups. If a user is in the Data Controller Admin Group , CLS rules DO NOT APPLY. CLS_LIBREF \u00b6 The library of the target table against which the security rule will be applied CLS_TABLE \u00b6 The target table against which the security rule will be applied CLS_VARIABLE_NM \u00b6 This is the name of the variable against which the security rule will be applied. Note that CLS_ACTIVE \u00b6 If you would like this rule to be applied, be sure this value is set to 1. CLS_HIDE \u00b6 This variable can be set to 1 to hide specific variables, which allows greater control over the EDIT screen in particular. CLS_SCOPE behaviour is impacted as follows: ALL - the variable will not be visible in either VIEW or EDIT. EDIT - the variable will not be visible. Cannot be applied to a primary key column . VIEW - the variable will not be visible. Can be applied to a primary key column. Simply omitting the row, or setting CLS_ACTIVE to 0, would result in the same behaviour. It is possible that a variable can have multiple values for CLS_HIDE, eg if a user is in multiple groups, or if different rules apply for different scopes. In this case, if the user is any group where this variable is NOT hidden, then it will be displayed. Example Config \u00b6 Example values as follows: CLS_SCOPE:$4 CLS_GROUP:$64 CLS_LIBREF:$8 CLS_TABLE:$32 CLS_VARIABLE_NM:$32 CLS_ACTIVE:8. CLS_HIDE:8. EDIT Group 1 MYLIB MYDS VAR_1 1 ALL Group 1 MYLIB MYDS VAR_2 1 ALL Group 2 MYLIB MYDS VAR_3 1 VIEW Group 1 MYLIB MYDS VAR_4 1 EDIT Group 1 MYLIB MYDS VAR_5 1 1 If a user is in Group 1, and viewing MYLIB.MYDS in EDIT mode, all columns will be visible but only the following columns will be editable: VAR_1 VAR_2 The user will be unable to add or delete rows. If the user is in both Group 1 AND Group 2, viewing MYLIB.MYDS in VIEW mode, only the following columns will be visible: VAR_2 VAR_3 VAR_4 Video Example \u00b6 This short video does a walkthrough of applying Column Level Security from end to end.","title":"Column Level Security"},{"location":"column-level-security/#column-level-security","text":"Column level security is implemented by mapping allowed columns to a list of SAS groups. In VIEW mode, only allowed columns are visible. In EDIT mode, allowed columns are editable - the remaining columns are read-only. Below is an example of an EDIT table with only one column enabled for editing: See also: Row Level Security .","title":"Column Level Security"},{"location":"column-level-security/#configuration","text":"The variables in MPE_COLUMN_LEVEL_SECURITY should be configured as follows:","title":"Configuration"},{"location":"column-level-security/#cls_scope","text":"Determines whether the rule applies to the VIEW page, the EDIT page, or ALL pages. The impact of the rule varies as follows:","title":"CLS_SCOPE"},{"location":"column-level-security/#view-scope","text":"When CLS_SCOPE in ('VIEW','ALL') then only the listed columns are visible (unless CLS_HIDE=1 )","title":"VIEW Scope"},{"location":"column-level-security/#edit-scope","text":"When CLS_SCOPE in ('EDIT','ALL') then only the listed columns are editable (the remaining columns are read-only, and visible). Furthermore: The user will be unable to ADD or DELETE records. Primary Key values are always read only Primary Key values cannot be hidden ( CLS_HIDE=1 will have no effect)","title":"EDIT Scope"},{"location":"column-level-security/#cls_group","text":"The SAS Group to which the rule applies. The user could also be a member of a DC group . If a user is in ANY of the groups, the columns will be restricted. If a user is in NONE of the groups, no restrictions apply (all columns available). If a user is in MULTIPLE groups, they will see all allowed columns across all groups. If a user is in the Data Controller Admin Group , CLS rules DO NOT APPLY.","title":"CLS_GROUP"},{"location":"column-level-security/#cls_libref","text":"The library of the target table against which the security rule will be applied","title":"CLS_LIBREF"},{"location":"column-level-security/#cls_table","text":"The target table against which the security rule will be applied","title":"CLS_TABLE"},{"location":"column-level-security/#cls_variable_nm","text":"This is the name of the variable against which the security rule will be applied. Note that","title":"CLS_VARIABLE_NM"},{"location":"column-level-security/#cls_active","text":"If you would like this rule to be applied, be sure this value is set to 1.","title":"CLS_ACTIVE"},{"location":"column-level-security/#cls_hide","text":"This variable can be set to 1 to hide specific variables, which allows greater control over the EDIT screen in particular. CLS_SCOPE behaviour is impacted as follows: ALL - the variable will not be visible in either VIEW or EDIT. EDIT - the variable will not be visible. Cannot be applied to a primary key column . VIEW - the variable will not be visible. Can be applied to a primary key column. Simply omitting the row, or setting CLS_ACTIVE to 0, would result in the same behaviour. It is possible that a variable can have multiple values for CLS_HIDE, eg if a user is in multiple groups, or if different rules apply for different scopes. In this case, if the user is any group where this variable is NOT hidden, then it will be displayed.","title":"CLS_HIDE"},{"location":"column-level-security/#example-config","text":"Example values as follows: CLS_SCOPE:$4 CLS_GROUP:$64 CLS_LIBREF:$8 CLS_TABLE:$32 CLS_VARIABLE_NM:$32 CLS_ACTIVE:8. CLS_HIDE:8. EDIT Group 1 MYLIB MYDS VAR_1 1 ALL Group 1 MYLIB MYDS VAR_2 1 ALL Group 2 MYLIB MYDS VAR_3 1 VIEW Group 1 MYLIB MYDS VAR_4 1 EDIT Group 1 MYLIB MYDS VAR_5 1 1 If a user is in Group 1, and viewing MYLIB.MYDS in EDIT mode, all columns will be visible but only the following columns will be editable: VAR_1 VAR_2 The user will be unable to add or delete rows. If the user is in both Group 1 AND Group 2, viewing MYLIB.MYDS in VIEW mode, only the following columns will be visible: VAR_2 VAR_3 VAR_4","title":"Example Config"},{"location":"column-level-security/#video-example","text":"This short video does a walkthrough of applying Column Level Security from end to end.","title":"Video Example"},{"location":"dc-overview/","text":"Data Controller for SAS\u00ae: Overview \u00b6 What does the Data Controller do? \u00b6 The Data Controller allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity. What is a Target Table? \u00b6 A Target Table is a physical table, such as a SAS dataset or a Table in a database. The attributes of this table (eg Primary Key, loadtype, library, SCD variables etc) will have been predefined by your administrator so that you can change the data in that table safely and easily. How does it work? \u00b6 From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (which may also be the editor, depending on configuration) reviews the changes and accepts / or rejects them. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded. Who is it for? \u00b6 There are 5 roles identified for users of the Data Controller: Viewer . A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (eg copy the url to share a table with a colleague). Editor . An editor makes changes to data in a table (add, modify, delete) and submits those changes to the approver(s) for acceptance. Approver . An approver accepts / rejects proposed changes to data under their control. If accepted, the change is applied to the target table. Auditor . An auditor has the ability to review the history of changes to a particular table. Administrator . An administrator has the ability to add new tables to the Data Controller, and to configure the security settings (at metadata group level) as required. What is a submission? \u00b6 The submission is the data that has been staged for approval. Note - submissions are never applied automatically! They must always be approved by 1 or more approvers first. The process of submission varies according to the type of submit. Web Submission \u00b6 When using the Web editor, a frontend check is made against the subset of data that was filtered for editing to see which rows are new / modified / marked deleted. Only those changed rows (from the extract) are submitted to the staging area. Excel Submission \u00b6 When importing an excel file, all rows are loaded into the web page. You have an option to edit those records. If you edit them, the original excel is discarded, and only changed rows are submitted (it becomes a web submission). If you hit SUBMIT immediately, then ALL rows are staged, and a copy of the excel file is uploaded for audit purposes. CSV submission \u00b6 A CSV upload bypasses the part where the records are loaded into the web page, and ALL rows are sent to the staging area directly. This makes it suitable for larger uploads. Edit Stage Approve Workflow \u00b6 Up to 500 rows can be edited (in the web editor) at one time. These edits are submitted to a staging area. After one or more approvals (acceptances) the changes are applied to the source table. Use Case Diagram \u00b6 There are five roles (Viewer, Editor, Approver, Auditor, Administrator) which correspond to 5 primary use cases (View Table, Edit Table, Approve Change, View Change History, Configure Table)","title":"Overview"},{"location":"dc-overview/#data-controller-for-sas-overview","text":"","title":"Data Controller for SAS\u00ae: Overview"},{"location":"dc-overview/#what-does-the-data-controller-do","text":"The Data Controller allows users to add, modify, and delete data. All changes are staged and approved before being applied to the target table. The review process, as well as using generic and repeatable code to perform updates, works to ensure data integrity.","title":"What does the Data Controller do?"},{"location":"dc-overview/#what-is-a-target-table","text":"A Target Table is a physical table, such as a SAS dataset or a Table in a database. The attributes of this table (eg Primary Key, loadtype, library, SCD variables etc) will have been predefined by your administrator so that you can change the data in that table safely and easily.","title":"What is a Target Table?"},{"location":"dc-overview/#how-does-it-work","text":"From the Editor tab, a user selects a library and table for editing. Data can then be edited directly, or a uploaded from a file. After submitting the change, the data is loaded to a secure staging area, and the approvers are notified. The approver (which may also be the editor, depending on configuration) reviews the changes and accepts / or rejects them. If accepted, the changes are applied to the target table by the system account, and the history of that change is recorded.","title":"How does it work?"},{"location":"dc-overview/#who-is-it-for","text":"There are 5 roles identified for users of the Data Controller: Viewer . A viewer uses the Data Controller as a means to explore data without risk of locking datasets. By using the Data Controller to view data, it also becomes possible to 'link' to data (eg copy the url to share a table with a colleague). Editor . An editor makes changes to data in a table (add, modify, delete) and submits those changes to the approver(s) for acceptance. Approver . An approver accepts / rejects proposed changes to data under their control. If accepted, the change is applied to the target table. Auditor . An auditor has the ability to review the history of changes to a particular table. Administrator . An administrator has the ability to add new tables to the Data Controller, and to configure the security settings (at metadata group level) as required.","title":"Who is it for?"},{"location":"dc-overview/#what-is-a-submission","text":"The submission is the data that has been staged for approval. Note - submissions are never applied automatically! They must always be approved by 1 or more approvers first. The process of submission varies according to the type of submit.","title":"What is a submission?"},{"location":"dc-overview/#web-submission","text":"When using the Web editor, a frontend check is made against the subset of data that was filtered for editing to see which rows are new / modified / marked deleted. Only those changed rows (from the extract) are submitted to the staging area.","title":"Web Submission"},{"location":"dc-overview/#excel-submission","text":"When importing an excel file, all rows are loaded into the web page. You have an option to edit those records. If you edit them, the original excel is discarded, and only changed rows are submitted (it becomes a web submission). If you hit SUBMIT immediately, then ALL rows are staged, and a copy of the excel file is uploaded for audit purposes.","title":"Excel Submission"},{"location":"dc-overview/#csv-submission","text":"A CSV upload bypasses the part where the records are loaded into the web page, and ALL rows are sent to the staging area directly. This makes it suitable for larger uploads.","title":"CSV submission"},{"location":"dc-overview/#edit-stage-approve-workflow","text":"Up to 500 rows can be edited (in the web editor) at one time. These edits are submitted to a staging area. After one or more approvals (acceptances) the changes are applied to the source table.","title":"Edit Stage Approve Workflow"},{"location":"dc-overview/#use-case-diagram","text":"There are five roles (Viewer, Editor, Approver, Auditor, Administrator) which correspond to 5 primary use cases (View Table, Edit Table, Approve Change, View Change History, Configure Table)","title":"Use Case Diagram"},{"location":"dc-userguide/","text":"Data Controller for SAS: User Guide \u00b6 Interface \u00b6 The Data Controller has 5 tabs, as follows: Viewer . This tab lets users view any table to which they have been granted access in metadata. They can also download the data as csv, excel, or as a SAS program (datalines). Primary key fields are coloured green. Editor . This tab enables users to add, modify or delete data. This can be done directly in the browser, or by uploading a CSV file. Values can also be copy-pasted from a spreadsheet. Once changes are ready, they can be submitted, with a corresponding reason. Submitted . This shows and editor the outstanding changes that have been submitted for approval (but have not yet been approved or rejected). Approvals . This shows an approver all their outstanding approval requests. History . This shows an auditor, or other interested party, what changes have been submitted for each table. Viewer \u00b6 Overview \u00b6 The viewer screen provides users with a raw view of underlying data. It is only possible to view tables that have been registered in metadata. Advantages of using the viewer (over client tools) for browsing data include: Ability to provide links to tables / filtered views of tables (just copy url) In the case of SAS datasets, prevent file locks from ocurring Ability to quickly download a CSV / Excel / SAS Cards program for that table Usage \u00b6 Choose a library, then a table, and click view to see the first 5000 rows. A filter option is provided should you wish to view a different section of rows. The Download button gives three options for obtaining the current view of data: 1) CSV. This provides a comma delimited file. 2) Excel. This provides a tab delimited file. 3) SAS. This provides a SAS program with data as datalines, so that the data can be rebuilt as a SAS table. Note - if the table is registered in Data Controller as being TXTEMPORAL (SCD2) then the download option will prefilter for the current records and removes the valid from / valid to variables. This makes the CSV a suitable format for subsequent DC file upload, if desired. Editor \u00b6 The Editor screen lets users who have been pre-authorised (via the DATACTRL.MPE_SECURITY table) to edit a particular table. A user selects a particular library, and table and then has 3 options: 1 - Filter . The user can filter before proceeding to perform edits. 2 - Upload . If you have a lot of data, you can upload it directly . The changes are then approved in the usual way. 3 - Edit . This is the main interface, data is displayed in tabular format. The first column is always \"Delete?\", as this allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted. The next set of columns are the Primary Key, and are shaded grey. If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend). For SCD2 type tables, the 'validity' fields are not shown. It is assumed that the user is always working with the current version of the data, and the view is filtered as such. After this, remaining columns are shown. Dates / datetime fields have appropriate datepickers. Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the DATACTRL.MPE_SELECTBOX table. New rows can be added using the right click context menu, or the 'Add Row' button. The data can also be sorted by clicking on the column headers. When ready to submit, hit the SUBMIT button and enter a reason for the change. The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen. If you are also an approver you can approve this change yourself. Special Missings \u00b6 Data Controller supports special missing numerics, ie - a single letter or underscore. These should be submitted without the leading period. The letters are not case sensitive. BiTemporal Tables \u00b6 The Data Controller only permits BiTemporal data uploads at a single point in time - so for convenience, when viewing data in the edit screen, only the most recent records are displayed. To edit earlier records, either use file upload, or apply a filter. Submitted \u00b6 This page shows a list of the changes you have submitted (that are not yet approved). Approvals \u00b6 This shows the list of changes that have been submitted to you (or your groups) for approval. History \u00b6 View the list of changes to each table, who made the change, when, etc.","title":"DC User Guide"},{"location":"dc-userguide/#data-controller-for-sas-user-guide","text":"","title":"Data Controller for SAS: User Guide"},{"location":"dc-userguide/#interface","text":"The Data Controller has 5 tabs, as follows: Viewer . This tab lets users view any table to which they have been granted access in metadata. They can also download the data as csv, excel, or as a SAS program (datalines). Primary key fields are coloured green. Editor . This tab enables users to add, modify or delete data. This can be done directly in the browser, or by uploading a CSV file. Values can also be copy-pasted from a spreadsheet. Once changes are ready, they can be submitted, with a corresponding reason. Submitted . This shows and editor the outstanding changes that have been submitted for approval (but have not yet been approved or rejected). Approvals . This shows an approver all their outstanding approval requests. History . This shows an auditor, or other interested party, what changes have been submitted for each table.","title":"Interface"},{"location":"dc-userguide/#viewer","text":"","title":"Viewer"},{"location":"dc-userguide/#overview","text":"The viewer screen provides users with a raw view of underlying data. It is only possible to view tables that have been registered in metadata. Advantages of using the viewer (over client tools) for browsing data include: Ability to provide links to tables / filtered views of tables (just copy url) In the case of SAS datasets, prevent file locks from ocurring Ability to quickly download a CSV / Excel / SAS Cards program for that table","title":"Overview"},{"location":"dc-userguide/#usage","text":"Choose a library, then a table, and click view to see the first 5000 rows. A filter option is provided should you wish to view a different section of rows. The Download button gives three options for obtaining the current view of data: 1) CSV. This provides a comma delimited file. 2) Excel. This provides a tab delimited file. 3) SAS. This provides a SAS program with data as datalines, so that the data can be rebuilt as a SAS table. Note - if the table is registered in Data Controller as being TXTEMPORAL (SCD2) then the download option will prefilter for the current records and removes the valid from / valid to variables. This makes the CSV a suitable format for subsequent DC file upload, if desired.","title":"Usage"},{"location":"dc-userguide/#editor","text":"The Editor screen lets users who have been pre-authorised (via the DATACTRL.MPE_SECURITY table) to edit a particular table. A user selects a particular library, and table and then has 3 options: 1 - Filter . The user can filter before proceeding to perform edits. 2 - Upload . If you have a lot of data, you can upload it directly . The changes are then approved in the usual way. 3 - Edit . This is the main interface, data is displayed in tabular format. The first column is always \"Delete?\", as this allows you to mark rows for deletion. Note that removing a row from display does not mark it for deletion! It simply means that this row is not part of the changeset being submitted. The next set of columns are the Primary Key, and are shaded grey. If the table has a surrogate / retained key, then it is the Business Key that is shown here (the RK field is calculated / updated at the backend). For SCD2 type tables, the 'validity' fields are not shown. It is assumed that the user is always working with the current version of the data, and the view is filtered as such. After this, remaining columns are shown. Dates / datetime fields have appropriate datepickers. Other fields may also have dropdowns to ensure entry of standard values, these can be configured in the DATACTRL.MPE_SELECTBOX table. New rows can be added using the right click context menu, or the 'Add Row' button. The data can also be sorted by clicking on the column headers. When ready to submit, hit the SUBMIT button and enter a reason for the change. The owners of the data are now alerted (so long as their email addresses are in metadata) with a link to the approve screen. If you are also an approver you can approve this change yourself.","title":"Editor"},{"location":"dc-userguide/#special-missings","text":"Data Controller supports special missing numerics, ie - a single letter or underscore. These should be submitted without the leading period. The letters are not case sensitive.","title":"Special Missings"},{"location":"dc-userguide/#bitemporal-tables","text":"The Data Controller only permits BiTemporal data uploads at a single point in time - so for convenience, when viewing data in the edit screen, only the most recent records are displayed. To edit earlier records, either use file upload, or apply a filter.","title":"BiTemporal Tables"},{"location":"dc-userguide/#submitted","text":"This page shows a list of the changes you have submitted (that are not yet approved).","title":"Submitted"},{"location":"dc-userguide/#approvals","text":"This shows the list of changes that have been submitted to you (or your groups) for approval.","title":"Approvals"},{"location":"dc-userguide/#history","text":"View the list of changes to each table, who made the change, when, etc.","title":"History"},{"location":"dcc-dates/","text":"Data Controller for SAS\u00ae - Dates & Datetimes \u00b6 Overview \u00b6 Dates & datetimes are actually stored as plain numerics in regular SAS tables. In order for the Data Controller to recognise these values as dates / datetimes a format must be applied. Supported date formats: DATE. DDMMYY. MMDDYY. YYMMDD. E8601DA. B8601DA. Supported datetime formats: DATETIME. Supported time formats: TIME. HHMM. In SAS 9, this format must also be present / updated in the metadata view of the (physical) table to be displayed properly. This can be done using DI Studio, or by running the following (template) code: proc metalib; omr (library= \"Your Library\" ); folder= \"/Shared Data/table storage location\" ; update_rule=( delete ) ; run; If you have other dates / datetimes / times you would like us to support, do get in touch !","title":"Dates / Datetimes"},{"location":"dcc-dates/#data-controller-for-sas-dates-datetimes","text":"","title":"Data Controller for SAS\u00ae - Dates & Datetimes"},{"location":"dcc-dates/#overview","text":"Dates & datetimes are actually stored as plain numerics in regular SAS tables. In order for the Data Controller to recognise these values as dates / datetimes a format must be applied. Supported date formats: DATE. DDMMYY. MMDDYY. YYMMDD. E8601DA. B8601DA. Supported datetime formats: DATETIME. Supported time formats: TIME. HHMM. In SAS 9, this format must also be present / updated in the metadata view of the (physical) table to be displayed properly. This can be done using DI Studio, or by running the following (template) code: proc metalib; omr (library= \"Your Library\" ); folder= \"/Shared Data/table storage location\" ; update_rule=( delete ) ; run; If you have other dates / datetimes / times you would like us to support, do get in touch !","title":"Overview"},{"location":"dcc-groups/","text":"Adding Groups \u00b6 Overview \u00b6 By default, Data Controller will work with the SAS Groups defined in Viya, Metadata, or SASjs Server. It is also possible to define custom groups with Data Controller itself - to do this simply add the user and group name (and optionally, a group description) in the DATACTRL.MPE_GROUPS table. Data Controller Admin Group \u00b6 When configuring Data Controller for the first time, a group is designated as the 'admin' group. This group has unrestricted access to Data Controller. To change this group, modify the %let dc_admin_group= entry in the settings program. To prevent others from changing this group, ensure the containing folder is write-protected!","title":"Groups"},{"location":"dcc-groups/#adding-groups","text":"","title":"Adding Groups"},{"location":"dcc-groups/#overview","text":"By default, Data Controller will work with the SAS Groups defined in Viya, Metadata, or SASjs Server. It is also possible to define custom groups with Data Controller itself - to do this simply add the user and group name (and optionally, a group description) in the DATACTRL.MPE_GROUPS table.","title":"Overview"},{"location":"dcc-groups/#data-controller-admin-group","text":"When configuring Data Controller for the first time, a group is designated as the 'admin' group. This group has unrestricted access to Data Controller. To change this group, modify the %let dc_admin_group= entry in the settings program. To prevent others from changing this group, ensure the containing folder is write-protected!","title":"Data Controller Admin Group"},{"location":"dcc-options/","text":"Data Controller for SAS\u00ae - Options \u00b6 The MPE_CONFIG table provides a number of system options, which apply to all users. The table may be re-purposed for other applications, so long as scopes beginning with \"DC_\" are avoided. Currently used scopes include: DC DC_CATALOG DC Scope \u00b6 DC_EMAIL_ALERTS \u00b6 Set to YES or NO to enable email alerts. This requires email options to be preconfigured (mail server etc). DC_MAXOBS_WEBEDIT \u00b6 By default, a maximum of 100 observations can be edited in the browser at one time. This number can be increased, but note that the following factors will impact performance: Number of configured Validations Browser type and version (works best in Chrome) Number (and size) of columns Speed of client machine (laptop/desktop) DC_RESTRICT_EDITRECORD \u00b6 Setting YES will prevent the EDIT RECORD dialog appearing in the EDIT screen by removing the \"Edit Row\" option in the right click menu, and the \"ADD RECORD\" button in the bottom left. Anything other than YES will mean that the modal is available. Default=NO DC_RESTRICT_VIEWER \u00b6 Set to YES to restrict the list of libraries and tables in VIEWER to only those explicitly set to VIEW in the MPE_SECURITY table. The default is NO (users can see all tables they already have permission to see). DC_VIEWLIB_CHECK \u00b6 Set to YES to enable library validity checking in viewLibs service. This means that on first load, SAS will attempt to open each library to see if it is possible to do so. This reduces the number of libraries in the list, but means that it is slow to load the first time around. The default is NO. DC_LOCALE \u00b6 Set to a locale (such as en_gb or en_be ) to override the system value (which may be derived from client browser settings). This feature is useful when importing ambiguous dates from CSV or Excel (eg 1/2/20 vs 2/1/20) as DC uses the anydtdtm. informats for import. Default=SYSTEM. Note If you have clients in different geographies loading excel in local formats, you can also address this issue by ensuring the locale of the windows user profile is not set to the default (eg English (United States) ). When leaving the DC_LOCALE as SYSTEM, the locale settings in SAS are not added or modified. DC_CATALOG Scope \u00b6 DC_IGNORELIBS \u00b6 When running the Refresh Data Catalog service, it is often that case the the process will fail due to being unable to assign a library. To avoid the need to resolve the connection issue elsewhere in SAS, you can simply exclude it from the Data Catalog, by including the LIBREF in this field (pipe-separated) DC_REVIEW Scope \u00b6 HISTORY_ROWS \u00b6 Number of rows to return for each HISTORY page. Default - 100. Increasing this will increase for all users. Using very large numbers here can result in a sluggish page load time. If you need large amounts of HISTORY data, it is generally better to extract it directly from the MPE_REVIEW table.","title":"Options"},{"location":"dcc-options/#data-controller-for-sas-options","text":"The MPE_CONFIG table provides a number of system options, which apply to all users. The table may be re-purposed for other applications, so long as scopes beginning with \"DC_\" are avoided. Currently used scopes include: DC DC_CATALOG","title":"Data Controller for SAS\u00ae - Options"},{"location":"dcc-options/#dc-scope","text":"","title":"DC Scope"},{"location":"dcc-options/#dc_email_alerts","text":"Set to YES or NO to enable email alerts. This requires email options to be preconfigured (mail server etc).","title":"DC_EMAIL_ALERTS"},{"location":"dcc-options/#dc_maxobs_webedit","text":"By default, a maximum of 100 observations can be edited in the browser at one time. This number can be increased, but note that the following factors will impact performance: Number of configured Validations Browser type and version (works best in Chrome) Number (and size) of columns Speed of client machine (laptop/desktop)","title":"DC_MAXOBS_WEBEDIT"},{"location":"dcc-options/#dc_restrict_editrecord","text":"Setting YES will prevent the EDIT RECORD dialog appearing in the EDIT screen by removing the \"Edit Row\" option in the right click menu, and the \"ADD RECORD\" button in the bottom left. Anything other than YES will mean that the modal is available. Default=NO","title":"DC_RESTRICT_EDITRECORD"},{"location":"dcc-options/#dc_restrict_viewer","text":"Set to YES to restrict the list of libraries and tables in VIEWER to only those explicitly set to VIEW in the MPE_SECURITY table. The default is NO (users can see all tables they already have permission to see).","title":"DC_RESTRICT_VIEWER"},{"location":"dcc-options/#dc_viewlib_check","text":"Set to YES to enable library validity checking in viewLibs service. This means that on first load, SAS will attempt to open each library to see if it is possible to do so. This reduces the number of libraries in the list, but means that it is slow to load the first time around. The default is NO.","title":"DC_VIEWLIB_CHECK"},{"location":"dcc-options/#dc_locale","text":"Set to a locale (such as en_gb or en_be ) to override the system value (which may be derived from client browser settings). This feature is useful when importing ambiguous dates from CSV or Excel (eg 1/2/20 vs 2/1/20) as DC uses the anydtdtm. informats for import. Default=SYSTEM. Note If you have clients in different geographies loading excel in local formats, you can also address this issue by ensuring the locale of the windows user profile is not set to the default (eg English (United States) ). When leaving the DC_LOCALE as SYSTEM, the locale settings in SAS are not added or modified.","title":"DC_LOCALE"},{"location":"dcc-options/#dc_catalog-scope","text":"","title":"DC_CATALOG Scope"},{"location":"dcc-options/#dc_ignorelibs","text":"When running the Refresh Data Catalog service, it is often that case the the process will fail due to being unable to assign a library. To avoid the need to resolve the connection issue elsewhere in SAS, you can simply exclude it from the Data Catalog, by including the LIBREF in this field (pipe-separated)","title":"DC_IGNORELIBS"},{"location":"dcc-options/#dc_review-scope","text":"","title":"DC_REVIEW Scope"},{"location":"dcc-options/#history_rows","text":"Number of rows to return for each HISTORY page. Default - 100. Increasing this will increase for all users. Using very large numbers here can result in a sluggish page load time. If you need large amounts of HISTORY data, it is generally better to extract it directly from the MPE_REVIEW table.","title":"HISTORY_ROWS"},{"location":"dcc-security/","text":"Data Controller for SAS\u00ae - Security \u00b6 Summary \u00b6 DC security is applied at the level of Table and Group. Permissions can only be set at group level. There are two parts to adding a user: 1 - Adding the user to the relevant group 2 - Ensuring that group has the appropriate access level in the MPE_SECURITY table For guidance with adding SAS users in SAS 9, see SAS Documentation . Details \u00b6 In order to surface a table to a new group, simply add a record to the DATACTRL.MPE_SECURITY table. The library.dataset value should go in the BASE_TABLE field, the level of access (either EDIT or APPROVE ) should go in the ACCESS_LEVEL field, and the exact name of the relevant metadata group should go in the SAS_GROUP field. The change should then be submitted, and approved, at which point the new security setting will be applied. ACCESS_LEVEL \u00b6 EDIT \u00b6 The EDIT permission determines which groups will be able to upload CSVs and submit changes via the web interface for that table. APPROVE \u00b6 The APPROVE permission determines which groups will be able to approve those changes, and hence enable the target table to be loaded. If you wish to have members of a particular group both edit AND approve, then two lines (one for each group) must be entered, per table. VIEW \u00b6 The default behaviour when installing Data Controller is that the viewer lets all SAS Users see all the tables that they are authorised to view in SAS. However there may be reasons to further restrict the tables in this component. There is a global setting that will disable ALL tables in VIEWER unless explicitly authorised - this is available in MPE_CONFIG. Set DC_RESTRICT_VIEWER=YES , submit, and approve. If authorising groups without this setting, it means that tables will be restricted only in that library (the rest will still be visible). Groups can be given VIEW access for all libraries or all tables within a library by using the keyword *ALL* instead of the libref / tablename. It's also worth being aware of the DC_VIEWLIB_CHECK option in MPE_CONFIG. When this is switched on, SAS will confirm that the library is valid and contains tables, before adding to the list. This can sometimes be slow (depending on your library configurations), hence disabled - but as the list is actually cached on frontend (until the next hard refresh) the impact may worth it. Determining Group Members \u00b6 Before adding a group to Data Controller, it helps to know the members of that group! A User navigator is available in both the SAS 9 and Viya version of Data Controller. You can navigate Users, Groups and Roles (roles are only visible in the SAS 9 version). This means you do not need SAS Management Console or SAS Environment Manager to manage Data Controller users. However you will need those tools for managing SAS Groups, unless you define your own groups in the MPE_GROUPS table.","title":"Security"},{"location":"dcc-security/#data-controller-for-sas-security","text":"","title":"Data Controller for SAS\u00ae - Security"},{"location":"dcc-security/#summary","text":"DC security is applied at the level of Table and Group. Permissions can only be set at group level. There are two parts to adding a user: 1 - Adding the user to the relevant group 2 - Ensuring that group has the appropriate access level in the MPE_SECURITY table For guidance with adding SAS users in SAS 9, see SAS Documentation .","title":"Summary"},{"location":"dcc-security/#details","text":"In order to surface a table to a new group, simply add a record to the DATACTRL.MPE_SECURITY table. The library.dataset value should go in the BASE_TABLE field, the level of access (either EDIT or APPROVE ) should go in the ACCESS_LEVEL field, and the exact name of the relevant metadata group should go in the SAS_GROUP field. The change should then be submitted, and approved, at which point the new security setting will be applied.","title":"Details"},{"location":"dcc-security/#access_level","text":"","title":"ACCESS_LEVEL"},{"location":"dcc-security/#edit","text":"The EDIT permission determines which groups will be able to upload CSVs and submit changes via the web interface for that table.","title":"EDIT"},{"location":"dcc-security/#approve","text":"The APPROVE permission determines which groups will be able to approve those changes, and hence enable the target table to be loaded. If you wish to have members of a particular group both edit AND approve, then two lines (one for each group) must be entered, per table.","title":"APPROVE"},{"location":"dcc-security/#view","text":"The default behaviour when installing Data Controller is that the viewer lets all SAS Users see all the tables that they are authorised to view in SAS. However there may be reasons to further restrict the tables in this component. There is a global setting that will disable ALL tables in VIEWER unless explicitly authorised - this is available in MPE_CONFIG. Set DC_RESTRICT_VIEWER=YES , submit, and approve. If authorising groups without this setting, it means that tables will be restricted only in that library (the rest will still be visible). Groups can be given VIEW access for all libraries or all tables within a library by using the keyword *ALL* instead of the libref / tablename. It's also worth being aware of the DC_VIEWLIB_CHECK option in MPE_CONFIG. When this is switched on, SAS will confirm that the library is valid and contains tables, before adding to the list. This can sometimes be slow (depending on your library configurations), hence disabled - but as the list is actually cached on frontend (until the next hard refresh) the impact may worth it.","title":"VIEW"},{"location":"dcc-security/#determining-group-members","text":"Before adding a group to Data Controller, it helps to know the members of that group! A User navigator is available in both the SAS 9 and Viya version of Data Controller. You can navigate Users, Groups and Roles (roles are only visible in the SAS 9 version). This means you do not need SAS Management Console or SAS Environment Manager to manage Data Controller users. However you will need those tools for managing SAS Groups, unless you define your own groups in the MPE_GROUPS table.","title":"Determining Group Members"},{"location":"dcc-selectbox/","text":"Data Controller for SAS\u00ae - Selectboxes \u00b6 Overview \u00b6 To ensure data validity and to improve user experience, it is possible to predesignate specific values for data entry. These appear to the user as a selectbox within the editor interface. Configuration \u00b6 Values are added by populating the DATACTRL.MPE_SELECTBOX table, eg below: BASE_LIBDS \u00b6 The library.dataset to which the selectbox needs to be applied BASE_COLUMN \u00b6 The column in which the selectbox values will be entered SELECTBOX_VALUE \u00b6 The actual values to be shown in the selectbox SELECTBOX_ORDER \u00b6 The order in which the selectbox values should be displayed SELECTBOX_TYPE \u00b6 Reserved for future use.","title":"Selectboxes"},{"location":"dcc-selectbox/#data-controller-for-sas-selectboxes","text":"","title":"Data Controller for SAS\u00ae - Selectboxes"},{"location":"dcc-selectbox/#overview","text":"To ensure data validity and to improve user experience, it is possible to predesignate specific values for data entry. These appear to the user as a selectbox within the editor interface.","title":"Overview"},{"location":"dcc-selectbox/#configuration","text":"Values are added by populating the DATACTRL.MPE_SELECTBOX table, eg below:","title":"Configuration"},{"location":"dcc-selectbox/#base_libds","text":"The library.dataset to which the selectbox needs to be applied","title":"BASE_LIBDS"},{"location":"dcc-selectbox/#base_column","text":"The column in which the selectbox values will be entered","title":"BASE_COLUMN"},{"location":"dcc-selectbox/#selectbox_value","text":"The actual values to be shown in the selectbox","title":"SELECTBOX_VALUE"},{"location":"dcc-selectbox/#selectbox_order","text":"The order in which the selectbox values should be displayed","title":"SELECTBOX_ORDER"},{"location":"dcc-selectbox/#selectbox_type","text":"Reserved for future use.","title":"SELECTBOX_TYPE"},{"location":"dcc-tables/","text":"Data Controller for SAS\u00ae - Adding Tables \u00b6 Overview \u00b6 Adding tables to the Data Controller is a matter of configuration, specifically the addition of a new record to the DATACTRL.MPE_TABLES table, and corresponding entries in the DATACTRL.MPE_SECURITY table. Note In order to surface the table to (non admin) users, appropriate groups should be configured as per security settings. MPE_TABLES Configuration Details \u00b6 Each table to be edited in the Data Controller is represented by one record in DATACTRL.MPE_TABLES . The fields should be populated as follows: LIBREF \u00b6 The libref of the table. If not pre-assigned, and the serverType is SAS 9 (EBI), DC will assign it at runtime using the first definition found in metadata, using this macro . DSN \u00b6 The dataset (table) name as visible when assigning a direct libref connection to LIBREF . If the target is a format catalog, it should have a \"-FC\" suffice (eg FORMATS-FC ). More info on formats here . NUM_OF_APPROVALS_REQUIRED \u00b6 This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, for example, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission. This is a required field. LOADTYPE \u00b6 The loadtype determines the nature of the update to be applied. Valid values are as follows: FORMAT_CAT. For updating Format Catalogs, the BUSKEY should be FMTNAME START . See formats . UPDATE. This is the most basic type, and any updates will happen 'in place'. Simply provide the primary key fields in the BUSKEY column. TXTEMPORAL. This signifies an SCD2 type load. For this type the validity fields (valid from, valid to) should be specified in the VAR_TXFROM and VAR_TXTO fields. The table itself should include VAR_TXFROM in the physical key. The remainder of the primary key fields (not including VAR_TXFROM ) should be specified in BUSKEY . BITEMPORAL. These tables have two time dimensions - a version history, and a business history. The version history (SCD2) fields should be specified in VAR_TXFROM and VAR_TXTO and the business history fields should be specified in VAR_BUSFROM and VAR_BUSTO . Both the VAR_TXFROM and VAR_BUSFROM fields should be in the physical key of the actual table, but should NOT be specified in the BUSKEY field. This is a required field. Note The support for BITEMPORAL loads is restricted, in the sense it is only possible to load data at a single point in time (no support for loading multiple business date ranges for a specific BUSKEY). The workaround is simply to load each date range separately. As a result of this restriction, the EDIT page will only show the latest business date range for each key. To modify earlier values, a filter should be applied. BUSKEY \u00b6 The business (natural) key of the table. For SCD2 / Bitemporal, this does NOT include the validity dates. For Retained / Surrogate key tables, this contains the actual surrogate key - the underlying fields that are used to create the surrogate key are specified in RK_UNDERLYING . This is a required field. VAR_TXFROM / VAR_TXTO \u00b6 The SCD2 type validity dates, representing the point in time at which the field was created ( VAR_TXFROM ) and when it was closed out ( VAR_TXTO ) from a change or deletion. If the record is active, the VAR_TXTO field would contain a high value. VAR_TXFROM is a part of the physical key of the underlying table. These fields should contain the NAME of the variables which contain the open / close timestamps in the underlying table. Leave blank if not required. VAR_BUSFROM / VAR_BUSTO \u00b6 The BITEMPORAL business dates which represent the reporting period to which the record is valid. Typically these contain date values (rather than datetime values). If variables are specified here, then the LOADTYPE should be BITEMPORAL . Leave blank if not required. VAR_PROCESSED \u00b6 Set the name of a variable (eg processed_dttm ) which should be given a current timestamp whenever the table is updated. Leave blank if not required. CLOSE_VARS \u00b6 By default, the Data Controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the absence of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The CLOSE_VARS are typically a subset of the BUSKEY fields. Leave blank if not required. PRE_EDIT_HOOK \u00b6 Hook script to execute prior to an edit being made. This allows data to be modified before being presented for editing. Leave blank if not required. SAS Developer Notes: Target dataset: work.OUT Filters will have been applied, and table sorted on BUSKEY Base libref.table or catalog variable: &orig_libds POST_EDIT_HOOK \u00b6 Hook script to execute after an edit has been made. Useful when there is a need to augment data, or perform advanced data quality checks prior to approval. Leave blank if not required. SAS Developer Notes: Target dataset: work.STAGING_DS Base libref.table or catalog variable: &orig_libds If your DQ check means that the program should not be submitted, then simply exit with &syscc > 4 . You can even set a message to go back to the user by using the mp_abort macro: %mp_abort(iftrue= (&syscc ne 0) /* if this condition is true, the process will exit */ ,msg=%str(YOUR MESSAGE GOES HERE) ) PRE_APPROVE_HOOK \u00b6 Hook script to execute before the approval diff is generated. It can be used to modify the values presented to an approver on the approve screen. This can be helpful in order to present the data in way that can be easily consumed by approvers. Leave blank if not required. SAS Developer Notes: Target dataset: work.STAGING_DS Base libref.table or catalog variable: &orig_libds POST_APPROVE_HOOK \u00b6 This hook script is %inc 'd after an approval is made. This is the most common type of hook script, and is useful for, say, running a SAS job after a mapping table is updated, or running a model after changing a parameter. Leave blank if not required. SAS Developer Notes: At the point of running this script, the data has already been loaded (successfully) to the target table. Therefore the target is typically the base libref.table (or format catalog) itself and can be referenced directly (YOURLIB.YOURDATASET), or using either of the following macro variable: &orig_libds &libref..&ds The staged table is also available, as work.STAGING_DS . If you are making changes to the target table as part of the hook, then in order to prevent contention from other users making concurrent edits, you are advised to \"LOCK\" and \"UNLOCK\" it using the mp_lockanytable macro: /* lock SOMELIB.SOMETABLE */ %mp_lockanytable(LOCK, lib=SOMELIB, ds=SOMETABLE, ref=Locking table to peform a post approve hook action ctl_ds=&mpelib..mpe_lockanytable ) /* do stuff */ proc sort data=somelib.sometable; run; /* unlock */ %mp_lockanytable(UNLOCK, lib=SOMELIB, ds=SOMETABLE, ctl_ds=&mpelib..mpe_lockanytable ) The SAS session will already contain the mp_lockanytable macro definition. SIGNOFF_COLS \u00b6 Used to determine a range (eg reporting month) to which a 'final version' can be marked. This allows a particular version of data to be marked as final, meaning that the data can continue to change afterwards (reports can simply query for the timestamp of the 'final' version of the data). Leave blank if not required. SIGNOFF_HOOK \u00b6 This hook script is %inc 'd after a 'final version' has been signed off. Leave blank if not required. NOTES \u00b6 Content entered here will be displayed to the approver on signoff. Not required, but recommended. RK_UNDERLYING \u00b6 For retained / surrogate keys, an auto-incrementing field is used to represent each unique record. In this case, the RK (integer) field itself should be added in the BUSKEY column, and the natural / underlying key should be added here. Leave blank unless using retained / surrogate keys. AUDIT_LIBDS \u00b6 If this field is blank (ie empty, missing), every change is captured in the MPE_AUDIT . This can result in large data volumes for frequently changing tables. Alternative options are: Enter a zero ( 0 ) to switch off audit logging completely Enter a library.dataset reference of an alternative audit table in which to capture the change history. For option 2, the base table structure can be generated using this macro: https://core.sasjs.io/mddl__dc__difftable_8sas_source.html . HOOK Scripts \u00b6 Data Controller allows SAS programs to be executed at certain points in the ingestion lifecycle, such as: Before an edit (to control the edit screen) After an edit (perform complex data quality) Before an approval (control the approve screen) After an approval (trigger downstream jobs with new data) The code is simply %include 'd at the relevant point during backend execution. The program may be: Physical, ie the full path to a .sas program on the physical server directory Logical, ie a Viya Job (SAS Drive), SAS 9 Stored Process (Metadata Folder) or SASJS Stored Program (SASjs Drive). If the entry ends in \".sas\" it is assumed to be a physical, filesystem file. Otherwise, the source code is extracted from SAS Drive or Metadata. To illustrate: Physical filesystem (ends in .sas): /opt/sas/code/myprogram.sas Logical filesystem: /Shared Data/stored_processes/mydatavalidator","title":"Tables"},{"location":"dcc-tables/#data-controller-for-sas-adding-tables","text":"","title":"Data Controller for SAS\u00ae - Adding Tables"},{"location":"dcc-tables/#overview","text":"Adding tables to the Data Controller is a matter of configuration, specifically the addition of a new record to the DATACTRL.MPE_TABLES table, and corresponding entries in the DATACTRL.MPE_SECURITY table. Note In order to surface the table to (non admin) users, appropriate groups should be configured as per security settings.","title":"Overview"},{"location":"dcc-tables/#mpe_tables-configuration-details","text":"Each table to be edited in the Data Controller is represented by one record in DATACTRL.MPE_TABLES . The fields should be populated as follows:","title":"MPE_TABLES Configuration Details"},{"location":"dcc-tables/#libref","text":"The libref of the table. If not pre-assigned, and the serverType is SAS 9 (EBI), DC will assign it at runtime using the first definition found in metadata, using this macro .","title":"LIBREF"},{"location":"dcc-tables/#dsn","text":"The dataset (table) name as visible when assigning a direct libref connection to LIBREF . If the target is a format catalog, it should have a \"-FC\" suffice (eg FORMATS-FC ). More info on formats here .","title":"DSN"},{"location":"dcc-tables/#num_of_approvals_required","text":"This is an integer representing the number of approvals required before a table is updated. This mechanism lets you insist on, for example, 2 or 3 approvals before sensitive data is updated following a submission. Note that only one rejection is ever necessary to remove the submission. This is a required field.","title":"NUM_OF_APPROVALS_REQUIRED"},{"location":"dcc-tables/#loadtype","text":"The loadtype determines the nature of the update to be applied. Valid values are as follows: FORMAT_CAT. For updating Format Catalogs, the BUSKEY should be FMTNAME START . See formats . UPDATE. This is the most basic type, and any updates will happen 'in place'. Simply provide the primary key fields in the BUSKEY column. TXTEMPORAL. This signifies an SCD2 type load. For this type the validity fields (valid from, valid to) should be specified in the VAR_TXFROM and VAR_TXTO fields. The table itself should include VAR_TXFROM in the physical key. The remainder of the primary key fields (not including VAR_TXFROM ) should be specified in BUSKEY . BITEMPORAL. These tables have two time dimensions - a version history, and a business history. The version history (SCD2) fields should be specified in VAR_TXFROM and VAR_TXTO and the business history fields should be specified in VAR_BUSFROM and VAR_BUSTO . Both the VAR_TXFROM and VAR_BUSFROM fields should be in the physical key of the actual table, but should NOT be specified in the BUSKEY field. This is a required field. Note The support for BITEMPORAL loads is restricted, in the sense it is only possible to load data at a single point in time (no support for loading multiple business date ranges for a specific BUSKEY). The workaround is simply to load each date range separately. As a result of this restriction, the EDIT page will only show the latest business date range for each key. To modify earlier values, a filter should be applied.","title":"LOADTYPE"},{"location":"dcc-tables/#buskey","text":"The business (natural) key of the table. For SCD2 / Bitemporal, this does NOT include the validity dates. For Retained / Surrogate key tables, this contains the actual surrogate key - the underlying fields that are used to create the surrogate key are specified in RK_UNDERLYING . This is a required field.","title":"BUSKEY"},{"location":"dcc-tables/#var_txfrom-var_txto","text":"The SCD2 type validity dates, representing the point in time at which the field was created ( VAR_TXFROM ) and when it was closed out ( VAR_TXTO ) from a change or deletion. If the record is active, the VAR_TXTO field would contain a high value. VAR_TXFROM is a part of the physical key of the underlying table. These fields should contain the NAME of the variables which contain the open / close timestamps in the underlying table. Leave blank if not required.","title":"VAR_TXFROM / VAR_TXTO"},{"location":"dcc-tables/#var_busfrom-var_busto","text":"The BITEMPORAL business dates which represent the reporting period to which the record is valid. Typically these contain date values (rather than datetime values). If variables are specified here, then the LOADTYPE should be BITEMPORAL . Leave blank if not required.","title":"VAR_BUSFROM / VAR_BUSTO"},{"location":"dcc-tables/#var_processed","text":"Set the name of a variable (eg processed_dttm ) which should be given a current timestamp whenever the table is updated. Leave blank if not required.","title":"VAR_PROCESSED"},{"location":"dcc-tables/#close_vars","text":"By default, the Data Controller will only process the records that are part of a changeset. This means that records should be explicity marked for deletion. But what if you are performing a reload of a monthly batch, and the absence of a record implies that it is no longer required? For this scenario, it is necessary to specify the range within a 'complete' load is expected. For instance, by reporting month, or month + product. When performing loads, the DC will then first extract a distinct list of values for this key and close them out in the target table, before performing the upload. The CLOSE_VARS are typically a subset of the BUSKEY fields. Leave blank if not required.","title":"CLOSE_VARS"},{"location":"dcc-tables/#pre_edit_hook","text":"Hook script to execute prior to an edit being made. This allows data to be modified before being presented for editing. Leave blank if not required. SAS Developer Notes: Target dataset: work.OUT Filters will have been applied, and table sorted on BUSKEY Base libref.table or catalog variable: &orig_libds","title":"PRE_EDIT_HOOK"},{"location":"dcc-tables/#post_edit_hook","text":"Hook script to execute after an edit has been made. Useful when there is a need to augment data, or perform advanced data quality checks prior to approval. Leave blank if not required. SAS Developer Notes: Target dataset: work.STAGING_DS Base libref.table or catalog variable: &orig_libds If your DQ check means that the program should not be submitted, then simply exit with &syscc > 4 . You can even set a message to go back to the user by using the mp_abort macro: %mp_abort(iftrue= (&syscc ne 0) /* if this condition is true, the process will exit */ ,msg=%str(YOUR MESSAGE GOES HERE) )","title":"POST_EDIT_HOOK"},{"location":"dcc-tables/#pre_approve_hook","text":"Hook script to execute before the approval diff is generated. It can be used to modify the values presented to an approver on the approve screen. This can be helpful in order to present the data in way that can be easily consumed by approvers. Leave blank if not required. SAS Developer Notes: Target dataset: work.STAGING_DS Base libref.table or catalog variable: &orig_libds","title":"PRE_APPROVE_HOOK"},{"location":"dcc-tables/#post_approve_hook","text":"This hook script is %inc 'd after an approval is made. This is the most common type of hook script, and is useful for, say, running a SAS job after a mapping table is updated, or running a model after changing a parameter. Leave blank if not required. SAS Developer Notes: At the point of running this script, the data has already been loaded (successfully) to the target table. Therefore the target is typically the base libref.table (or format catalog) itself and can be referenced directly (YOURLIB.YOURDATASET), or using either of the following macro variable: &orig_libds &libref..&ds The staged table is also available, as work.STAGING_DS . If you are making changes to the target table as part of the hook, then in order to prevent contention from other users making concurrent edits, you are advised to \"LOCK\" and \"UNLOCK\" it using the mp_lockanytable macro: /* lock SOMELIB.SOMETABLE */ %mp_lockanytable(LOCK, lib=SOMELIB, ds=SOMETABLE, ref=Locking table to peform a post approve hook action ctl_ds=&mpelib..mpe_lockanytable ) /* do stuff */ proc sort data=somelib.sometable; run; /* unlock */ %mp_lockanytable(UNLOCK, lib=SOMELIB, ds=SOMETABLE, ctl_ds=&mpelib..mpe_lockanytable ) The SAS session will already contain the mp_lockanytable macro definition.","title":"POST_APPROVE_HOOK"},{"location":"dcc-tables/#signoff_cols","text":"Used to determine a range (eg reporting month) to which a 'final version' can be marked. This allows a particular version of data to be marked as final, meaning that the data can continue to change afterwards (reports can simply query for the timestamp of the 'final' version of the data). Leave blank if not required.","title":"SIGNOFF_COLS"},{"location":"dcc-tables/#signoff_hook","text":"This hook script is %inc 'd after a 'final version' has been signed off. Leave blank if not required.","title":"SIGNOFF_HOOK"},{"location":"dcc-tables/#notes","text":"Content entered here will be displayed to the approver on signoff. Not required, but recommended.","title":"NOTES"},{"location":"dcc-tables/#rk_underlying","text":"For retained / surrogate keys, an auto-incrementing field is used to represent each unique record. In this case, the RK (integer) field itself should be added in the BUSKEY column, and the natural / underlying key should be added here. Leave blank unless using retained / surrogate keys.","title":"RK_UNDERLYING"},{"location":"dcc-tables/#audit_libds","text":"If this field is blank (ie empty, missing), every change is captured in the MPE_AUDIT . This can result in large data volumes for frequently changing tables. Alternative options are: Enter a zero ( 0 ) to switch off audit logging completely Enter a library.dataset reference of an alternative audit table in which to capture the change history. For option 2, the base table structure can be generated using this macro: https://core.sasjs.io/mddl__dc__difftable_8sas_source.html .","title":"AUDIT_LIBDS"},{"location":"dcc-tables/#hook-scripts","text":"Data Controller allows SAS programs to be executed at certain points in the ingestion lifecycle, such as: Before an edit (to control the edit screen) After an edit (perform complex data quality) Before an approval (control the approve screen) After an approval (trigger downstream jobs with new data) The code is simply %include 'd at the relevant point during backend execution. The program may be: Physical, ie the full path to a .sas program on the physical server directory Logical, ie a Viya Job (SAS Drive), SAS 9 Stored Process (Metadata Folder) or SASJS Stored Program (SASjs Drive). If the entry ends in \".sas\" it is assumed to be a physical, filesystem file. Otherwise, the source code is extracted from SAS Drive or Metadata. To illustrate: Physical filesystem (ends in .sas): /opt/sas/code/myprogram.sas Logical filesystem: /Shared Data/stored_processes/mydatavalidator","title":"HOOK Scripts"},{"location":"dcc-validations/","text":"Data Controller for SAS\u00ae - DQ Validations \u00b6 Overview \u00b6 Quality in, Quality out! Data Controller lets you enforce quality checks at the point of data entry, both directly via the web interface and also via Excel uploads. Default Checks \u00b6 By default, the following frontend rules are always applied: Length checking per target table variable lengths Type checking per target table datatypes (Character, Numeric, Date, Time, Datetime) Not Null check per target table constraints Primary Key checking per business key defined in MPE_TABLES It is possible to configure a number of other rules by updating the MPE_VALIDATIONS table. Simply set the BASE_LIB , BASE_DS and BASE_COL values, and ensure RULE_ACTIVE=1 for it to be applied. Configurable Checks \u00b6 Check back frequently as we plan to keep growing this list of checks. Rule Type Example Value Description CASE UPCASE Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE NOTNULL (defaultval) Will prevent submission if null values are present. Optional - provide a default value. MINVAL 1 Defines a minimum value for a numeric cell MAXVAL 1000000 Defines a maximum value for a numeric cell HARDSELECT sashelp.class.name A distinct list of values (max 1000) are taken from this library.member.column reference, and the value must be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table. SOFTSELECT dcdemo.mpe_tables.libref A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table. HARDSELECT_HOOK /logical/folder/stpname A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User provided values must be in this list. Cannot be used alongside a SOFTSELECT_HOOK. SOFTSELECT_HOOK /physical/path/program.sas A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User-provided values may (or may not) be in this list. Cannot be used alongside a HARDSELECT_HOOK. Dropdowns \u00b6 There are now actually FIVE places where you can configure dropdowns! The MPE_SELECTBOX table The HARDSELECT validation (library.member.column reference) The SOFTSELECT validation (library.member.column reference) The HARDSELECT_HOOK validation (SAS Program) The SOFTSELECT_HOOK validation (SAS Program) How do these inter-operate? Well - if you have values in MPE_SELECTBOX and/or HARDSELECT / SOFTSELECT tables, they will be merged together, and served in ADDITION to the values provided by any HOOK program. Dropdowns are SOFT by default, unless a HARD rule is present. Data Controller will not let you submit both a HARDSELECT_HOOK and a SOFTSELECT_HOOK on the same variable.","title":"Validations"},{"location":"dcc-validations/#data-controller-for-sas-dq-validations","text":"","title":"Data Controller for SAS\u00ae - DQ Validations"},{"location":"dcc-validations/#overview","text":"Quality in, Quality out! Data Controller lets you enforce quality checks at the point of data entry, both directly via the web interface and also via Excel uploads.","title":"Overview"},{"location":"dcc-validations/#default-checks","text":"By default, the following frontend rules are always applied: Length checking per target table variable lengths Type checking per target table datatypes (Character, Numeric, Date, Time, Datetime) Not Null check per target table constraints Primary Key checking per business key defined in MPE_TABLES It is possible to configure a number of other rules by updating the MPE_VALIDATIONS table. Simply set the BASE_LIB , BASE_DS and BASE_COL values, and ensure RULE_ACTIVE=1 for it to be applied.","title":"Default Checks"},{"location":"dcc-validations/#configurable-checks","text":"Check back frequently as we plan to keep growing this list of checks. Rule Type Example Value Description CASE UPCASE Will enforce the case of cell values. Valid values: UPCASE, LOWCASE, PROPCASE NOTNULL (defaultval) Will prevent submission if null values are present. Optional - provide a default value. MINVAL 1 Defines a minimum value for a numeric cell MAXVAL 1000000 Defines a maximum value for a numeric cell HARDSELECT sashelp.class.name A distinct list of values (max 1000) are taken from this library.member.column reference, and the value must be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table. SOFTSELECT dcdemo.mpe_tables.libref A distinct list of values (max 1000) are taken from this library.member.column reference, and the user-provided value may (or may not) be in this list. This list may be supplemented by entries in the MPE_SELECTBOX table. HARDSELECT_HOOK /logical/folder/stpname A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User provided values must be in this list. Cannot be used alongside a SOFTSELECT_HOOK. SOFTSELECT_HOOK /physical/path/program.sas A SAS service (STP or Viya Job) or a path to a SAS program on the filesystem. User-provided values may (or may not) be in this list. Cannot be used alongside a HARDSELECT_HOOK.","title":"Configurable Checks"},{"location":"dcc-validations/#dropdowns","text":"There are now actually FIVE places where you can configure dropdowns! The MPE_SELECTBOX table The HARDSELECT validation (library.member.column reference) The SOFTSELECT validation (library.member.column reference) The HARDSELECT_HOOK validation (SAS Program) The SOFTSELECT_HOOK validation (SAS Program) How do these inter-operate? Well - if you have values in MPE_SELECTBOX and/or HARDSELECT / SOFTSELECT tables, they will be merged together, and served in ADDITION to the values provided by any HOOK program. Dropdowns are SOFT by default, unless a HARD rule is present. Data Controller will not let you submit both a HARDSELECT_HOOK and a SOFTSELECT_HOOK on the same variable.","title":"Dropdowns"},{"location":"dci-deploysas9/","text":"SAS 9 Deployment \u00b6 Deployment Process \u00b6 There are two ways to deploy Data Controller on SAS 9: Full Deployment (preferred) Streaming (for quick demos) Full Deployment \u00b6 1 - Deploy Stored Processes \u00b6 The Stored Processes are deployed using a SAS Program. This should be executed using an account that has WRITE METADATA (WM) permissions to the necessary root folder ( appLoc ) in metadata. %let appLoc=/Shared Data/apps/DataController; /* CHANGE THIS!! */ filename dc url \"https://git.4gl.io/dc/deploy/-/raw/main/s9_noweb.sas\" ; %inc dc; If you don't have internet access from SAS, download this file and change the compiled_apploc on line 2: You can also change the serverName here, which is necessary if you are using any other logical server than SASApp . 2 - Deploy the Frontend \u00b6 The Data Controller frontend comes pre-built, and ready to deploy to the root of the SAS Web Server (mid-tier). Deploy as follows: Download the zip file from: https://git.4gl.io/dc/deploy/-/raw/main/frontend.zip Unzip and place in the htdocs folder of your SAS Web Server - typically !SASCONFIG/LevX/Web/WebServer/htdocs . Open the index.html file and update the values for appLoc (per SAS code above) and serverType (to SAS9 ). You can now open the app at https://YOURWEBSERVER/unzippedfoldername and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment. 3 - Run the Configurator \u00b6 When opening Data Controller for the first time, a configuration screen is presented. Be sure to log in with an account that has WRITE METADATA (WM) on the following metadata folders: services/admin - so the configurator STP can be deleted after being run services/common - so the Data_Controller_Settings STP can be updated Data - so the library and tables can be registered (using proc metalib) There are two things to configure: Path to the designated physical staging area. Make sure that the SAS Spawned Server account (eg sassrv ) has WRITE access to this location. Admin Group. \u26a0\ufe0f Note that anyone in this group will have unrestricted access to Data Controller! \u26a0\ufe0f \"Unrestricted access\" is provided by code logic. Post installation, Data Controller will never update nor modify metadata. Note If you do not see any groups, then it is possible your Stored Process is running from a different metadata repository to the location of your SAS users (eg Foundation). To fix this, update the services/admin/configurator STP with this code: %let dc_repo_users=YOUUSERRMETAREPO; After you click submit, the Stored Process will run, configure the staging area and create the library tables (as datasets). You will then be presented with three further links: Refresh Data Catalog. Run this to scan all available datasets and update the catalog. Refresh Table Metadata. Run this to update the table-level data lineage. Launch. Currently this feature only works for streaming apps - just refresh the page for a full deployment. Streaming \u00b6 The streaming approach is optimised for rapid deployment, and works by bundling the frontend into metadata. This is a highly inefficient way to serve web content, and thus should only really be used for demos / evaluation purposes. Deployment is very easy - just run the SAS code below (after changing the appLoc ): %let appLoc=/Shared Data/apps/DataController; /* CHANGE THIS!! */ filename dc url \"https://git.4gl.io/dc/deploy/-/raw/main/s9.sas\" ; %inc dc; If you don't have internet access from your SAS environment, you can also download the file directly and modify the appLoc on line 2, as follows: After that, continue to the configuration as described above. Deployment Diagram \u00b6 A Full Deployment of Data Controller for SAS 9 consists of: Frontend on the web server Stored Processes (+ Library & Table definitions) in metadata Staging Area on the physical filesystem Database or SAS Base library The below areas of the SAS platform are modified when deploying Data Controller: Client Device \u00b6 Nothing needs to be deployed or modified on the client device. We support a wide range of browsers (the same as SAS). Browsers make requests to the SAS Web Server, and will cache assets such as JS, CSS and images. Some items (such as dropdowns) are kept in local storage to improve responsiveness. SAS Mid Tier \u00b6 A single index.html file plus several CSS / JS / image files are served from a subfolder in the static content area SAS Web Server. This is served up by the existing SAS Web Server, no additional server (running) process is required. If you are running more than one web server, you will need to deploy to them all. SAS Application Server \u00b6 Given the enhanced permissions needed of the system account, a dedicated / secured STP instance is recommended as described here . All deployments of Data Controller also make use of a physical staging directory. This is used to store staged data, logs, plus CSV and Excel files as uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account (eg sassrv ) requires access to this directory. A typical small deployment will grow by a 10-20 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 1 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size. SAS Metadata Server \u00b6 The items deployed to metadata include: Folder tree Stored Processes Library Object & tables All SAS code is embedded in Stored Processes (so there is no need to deploy programs to the file system, no SASAUTOs). There is no use of X commands, no use of external internet access, full LOCKDOWN is supported. After the installation process (which updates public/settings and removes the admin/makedata STP), there are no write actions performed against metadata. Databases \u00b6 We strongly recommend that the Data Controller configuration tables are stored in a database for concurrency reasons. We have customers in production using Oracle, Postgres, Netezza, SQL Server to name a few. Contact us for support with DDL and migration steps for your chosen vendor. Note Data Controller does NOT modify schemas! It will not create or drop tables, or add/modify columns or attributes. Only data values (not the model) can be modified using this tool. To caveat the above - it is also quite common for customers to use a BASE engine library. Data Controller ships with mechananisms to handle locking (internally) but it cannot handle external contentions, such as those caused when end users open datasets directly, eg with Enterprise Guide or Base SAS.","title":"SAS 9 Deploy"},{"location":"dci-deploysas9/#sas-9-deployment","text":"","title":"SAS 9 Deployment"},{"location":"dci-deploysas9/#deployment-process","text":"There are two ways to deploy Data Controller on SAS 9: Full Deployment (preferred) Streaming (for quick demos)","title":"Deployment Process"},{"location":"dci-deploysas9/#full-deployment","text":"","title":"Full Deployment"},{"location":"dci-deploysas9/#1-deploy-stored-processes","text":"The Stored Processes are deployed using a SAS Program. This should be executed using an account that has WRITE METADATA (WM) permissions to the necessary root folder ( appLoc ) in metadata. %let appLoc=/Shared Data/apps/DataController; /* CHANGE THIS!! */ filename dc url \"https://git.4gl.io/dc/deploy/-/raw/main/s9_noweb.sas\" ; %inc dc; If you don't have internet access from SAS, download this file and change the compiled_apploc on line 2: You can also change the serverName here, which is necessary if you are using any other logical server than SASApp .","title":"1 - Deploy Stored Processes"},{"location":"dci-deploysas9/#2-deploy-the-frontend","text":"The Data Controller frontend comes pre-built, and ready to deploy to the root of the SAS Web Server (mid-tier). Deploy as follows: Download the zip file from: https://git.4gl.io/dc/deploy/-/raw/main/frontend.zip Unzip and place in the htdocs folder of your SAS Web Server - typically !SASCONFIG/LevX/Web/WebServer/htdocs . Open the index.html file and update the values for appLoc (per SAS code above) and serverType (to SAS9 ). You can now open the app at https://YOURWEBSERVER/unzippedfoldername and follow the configuration steps (DC Physical Location and Admin Group) to complete deployment.","title":"2 - Deploy the Frontend"},{"location":"dci-deploysas9/#3-run-the-configurator","text":"When opening Data Controller for the first time, a configuration screen is presented. Be sure to log in with an account that has WRITE METADATA (WM) on the following metadata folders: services/admin - so the configurator STP can be deleted after being run services/common - so the Data_Controller_Settings STP can be updated Data - so the library and tables can be registered (using proc metalib) There are two things to configure: Path to the designated physical staging area. Make sure that the SAS Spawned Server account (eg sassrv ) has WRITE access to this location. Admin Group. \u26a0\ufe0f Note that anyone in this group will have unrestricted access to Data Controller! \u26a0\ufe0f \"Unrestricted access\" is provided by code logic. Post installation, Data Controller will never update nor modify metadata. Note If you do not see any groups, then it is possible your Stored Process is running from a different metadata repository to the location of your SAS users (eg Foundation). To fix this, update the services/admin/configurator STP with this code: %let dc_repo_users=YOUUSERRMETAREPO; After you click submit, the Stored Process will run, configure the staging area and create the library tables (as datasets). You will then be presented with three further links: Refresh Data Catalog. Run this to scan all available datasets and update the catalog. Refresh Table Metadata. Run this to update the table-level data lineage. Launch. Currently this feature only works for streaming apps - just refresh the page for a full deployment.","title":"3 - Run the Configurator"},{"location":"dci-deploysas9/#streaming","text":"The streaming approach is optimised for rapid deployment, and works by bundling the frontend into metadata. This is a highly inefficient way to serve web content, and thus should only really be used for demos / evaluation purposes. Deployment is very easy - just run the SAS code below (after changing the appLoc ): %let appLoc=/Shared Data/apps/DataController; /* CHANGE THIS!! */ filename dc url \"https://git.4gl.io/dc/deploy/-/raw/main/s9.sas\" ; %inc dc; If you don't have internet access from your SAS environment, you can also download the file directly and modify the appLoc on line 2, as follows: After that, continue to the configuration as described above.","title":"Streaming"},{"location":"dci-deploysas9/#deployment-diagram","text":"A Full Deployment of Data Controller for SAS 9 consists of: Frontend on the web server Stored Processes (+ Library & Table definitions) in metadata Staging Area on the physical filesystem Database or SAS Base library The below areas of the SAS platform are modified when deploying Data Controller:","title":"Deployment Diagram"},{"location":"dci-deploysas9/#client-device","text":"Nothing needs to be deployed or modified on the client device. We support a wide range of browsers (the same as SAS). Browsers make requests to the SAS Web Server, and will cache assets such as JS, CSS and images. Some items (such as dropdowns) are kept in local storage to improve responsiveness.","title":"Client Device"},{"location":"dci-deploysas9/#sas-mid-tier","text":"A single index.html file plus several CSS / JS / image files are served from a subfolder in the static content area SAS Web Server. This is served up by the existing SAS Web Server, no additional server (running) process is required. If you are running more than one web server, you will need to deploy to them all.","title":"SAS Mid Tier"},{"location":"dci-deploysas9/#sas-application-server","text":"Given the enhanced permissions needed of the system account, a dedicated / secured STP instance is recommended as described here . All deployments of Data Controller also make use of a physical staging directory. This is used to store staged data, logs, plus CSV and Excel files as uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account (eg sassrv ) requires access to this directory. A typical small deployment will grow by a 10-20 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 1 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size.","title":"SAS Application Server"},{"location":"dci-deploysas9/#sas-metadata-server","text":"The items deployed to metadata include: Folder tree Stored Processes Library Object & tables All SAS code is embedded in Stored Processes (so there is no need to deploy programs to the file system, no SASAUTOs). There is no use of X commands, no use of external internet access, full LOCKDOWN is supported. After the installation process (which updates public/settings and removes the admin/makedata STP), there are no write actions performed against metadata.","title":"SAS Metadata Server"},{"location":"dci-deploysas9/#databases","text":"We strongly recommend that the Data Controller configuration tables are stored in a database for concurrency reasons. We have customers in production using Oracle, Postgres, Netezza, SQL Server to name a few. Contact us for support with DDL and migration steps for your chosen vendor. Note Data Controller does NOT modify schemas! It will not create or drop tables, or add/modify columns or attributes. Only data values (not the model) can be modified using this tool. To caveat the above - it is also quite common for customers to use a BASE engine library. Data Controller ships with mechananisms to handle locking (internally) but it cannot handle external contentions, such as those caused when end users open datasets directly, eg with Enterprise Guide or Base SAS.","title":"Databases"},{"location":"dci-deploysasviya/","text":"SAS Viya Deployment \u00b6 Overview \u00b6 Data Controller for SAS Viya consists of a frontend, a set of Job Execution Services, a staging area, a Compute Context, and a database library. The library can be a SAS Base engine if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS. A database that supports concurrent access is highly recommended. Prerequisites \u00b6 System Account \u00b6 Data Controller makes use of a system account for performing backend data updates and writing to the staging area. This needs to be provisioned in advance using the Viya admin-cli. The process is well described here: https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992 Database \u00b6 Whilst we do recommend that Data Controller configuration tables are stored in a database for concurrency reasons, it is also possible to use a BASE engine library, which is adequate if you only have a few users. To migrate the control library to a database, first perform a regular deployment, and afterwards you can generate the DDL and update the settings file.. Make sure the system account (see above) has full read / write access. Note \"Modify schema\" privileges are not required. Staging Directory \u00b6 All deployments of Data Controller make use of a physical staging directory. This is used to store logs, as well as CSV and Excel files uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account requires access to this directory. A typical small deployment will grow by a 5-10 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 0.5 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size. Deployment Diagram \u00b6 The below areas of the SAS Viya platform are modified when deploying Data Controller: Deployment \u00b6 Data Controller deployment is split between 3 deployment types: Demo version Full Version (manual deploy) Full Version (automated deploy) There are several parts to this proces: Create the Compute Context Deploy Frontend Prepare the database and update settings (optional) Update the Compute Context autoexec Create Compute Context \u00b6 The Viya Compute context is used to spawn the Job Execution Services - such that those services may run under the specified system account, with a particular autoexec. We strongly recommend a dedicated compute context for running Data Controller. The setup requires an Administrator account. Log onto SASEnvironment Manager, select Contexts, View Compute Contexts, and click the Create icon. In the New Compute Context dialog, enter the following attributes: Context Name Launcher Context Attribute pairs: reuseServerProcesses: true runServerAs: {{the account set up earlier }} Save and exit Note XCMD is NOT required to use Data Controller. Deploy frontend \u00b6 Unzip the frontend into your chosen directory (eg /var/www/html/DataController ) on the SAS Web Server. Open index.html and update the following inside dcAdapterSettings : appLoc - this should point to the root folder on SAS Drive where you would like the Job Execution services to be created. This folder should initially, NOT exist (if it is found, the backend will not be deployed) contextName - here you should put the name of the compute context you created in the previous step. dcPath - the physical location on the filesystem to be used for staged data. This is only used at deployment time, it can be configured later in $(appLoc)/services/settings.sas or in the autoexec if used. adminGroup - the name of an existing group, which should have unrestricted access to Data Controller. This is only used at deployment time, it can be configured later in $(appLoc)/services/settings.sas or in the autoexec if used. servertype - should be SASVIYA debug - can stay as false for performance, but could be switched to true for debugging startup issues useComputeApi - use true for best performance. Now, open https://YOURSERVER/DataController (using whichever subfolder you deployed to above) using an account that has the SAS privileges to write to the appLoc location. You will be presented with a deployment screen like the one below. Be sure to check the \"Recreate Database\" option and then click the \"Deploy\" button. Your services are deployed! And the app is operational, albeit still a little sluggish, as every single request is using the APIs to fetch the content of the $(appLoc)/services/settings.sas file. To improve responsiveness by another 700ms we recommend you follow the steps in Update Compute Context Autoexec below. Deploy Database \u00b6 If you have a lot of users, such that concurrency (locked datasets) becomes an issue, you might consider migrating the control library to a database. The first part to this is generating the DDL (and inserts). For this, use the DDL exporter as described here . If you need a flavour of DDL that is not yet supported, contact us . Step 2 is simply to run this DDL in your preferred database. Step 3 is to update the library definition in the $(appLoc)/services/settings.sas file using SAS Studio. Update Compute Context Autoexec \u00b6 First, open the $(appLoc)/services/settings.sas file in SAS Studio, and copy the code. Then, open SASEnvironment Manager, select Contexts, View Compute Contexts, and open the context we created earlier. Switch to the Advanced tab and paste in the SAS code copied from SAS Studio above. It will look similar to: %let DC_LIBREF=DCDBVIYA; %let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}}; %let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}}; libname &dc_libref {{YOUR DC DATABASE}}; To explain each of these lines: DC_LIBREF can be any valid 8 character libref. DC_ADMIN_GROUP is the name of the group which will have unrestricted access to Data Controller DC_STAGING_AREA should point to the location on the filesystem where the staging files and logs are be stored The final libname statement can also be configured to point at a database instead of a BASE engine directory (contact us for DDL) If you have additional libraries that you would like to use in Data Controller, they should also be defined here.","title":"SAS Viya Deploy"},{"location":"dci-deploysasviya/#sas-viya-deployment","text":"","title":"SAS Viya Deployment"},{"location":"dci-deploysasviya/#overview","text":"Data Controller for SAS Viya consists of a frontend, a set of Job Execution Services, a staging area, a Compute Context, and a database library. The library can be a SAS Base engine if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS. A database that supports concurrent access is highly recommended.","title":"Overview"},{"location":"dci-deploysasviya/#prerequisites","text":"","title":"Prerequisites"},{"location":"dci-deploysasviya/#system-account","text":"Data Controller makes use of a system account for performing backend data updates and writing to the staging area. This needs to be provisioned in advance using the Viya admin-cli. The process is well described here: https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-3-5-Compute-Server-Service-Accounts/ta-p/620992","title":"System Account"},{"location":"dci-deploysasviya/#database","text":"Whilst we do recommend that Data Controller configuration tables are stored in a database for concurrency reasons, it is also possible to use a BASE engine library, which is adequate if you only have a few users. To migrate the control library to a database, first perform a regular deployment, and afterwards you can generate the DDL and update the settings file.. Make sure the system account (see above) has full read / write access. Note \"Modify schema\" privileges are not required.","title":"Database"},{"location":"dci-deploysasviya/#staging-directory","text":"All deployments of Data Controller make use of a physical staging directory. This is used to store logs, as well as CSV and Excel files uploaded by end users. This directory should NOT be accessible by end users - only the SAS system account requires access to this directory. A typical small deployment will grow by a 5-10 mb each month. A very large enterprise customer, with 100 or more editors, might generate up to 0.5 GB or so per month, depending on the size and frequency of the Excel EUCs and CSVs being uploaded. Web modifications are restricted only to modified rows, so are typically just a few kb in size.","title":"Staging Directory"},{"location":"dci-deploysasviya/#deployment-diagram","text":"The below areas of the SAS Viya platform are modified when deploying Data Controller:","title":"Deployment Diagram"},{"location":"dci-deploysasviya/#deployment","text":"Data Controller deployment is split between 3 deployment types: Demo version Full Version (manual deploy) Full Version (automated deploy) There are several parts to this proces: Create the Compute Context Deploy Frontend Prepare the database and update settings (optional) Update the Compute Context autoexec","title":"Deployment"},{"location":"dci-deploysasviya/#create-compute-context","text":"The Viya Compute context is used to spawn the Job Execution Services - such that those services may run under the specified system account, with a particular autoexec. We strongly recommend a dedicated compute context for running Data Controller. The setup requires an Administrator account. Log onto SASEnvironment Manager, select Contexts, View Compute Contexts, and click the Create icon. In the New Compute Context dialog, enter the following attributes: Context Name Launcher Context Attribute pairs: reuseServerProcesses: true runServerAs: {{the account set up earlier }} Save and exit Note XCMD is NOT required to use Data Controller.","title":"Create Compute Context"},{"location":"dci-deploysasviya/#deploy-frontend","text":"Unzip the frontend into your chosen directory (eg /var/www/html/DataController ) on the SAS Web Server. Open index.html and update the following inside dcAdapterSettings : appLoc - this should point to the root folder on SAS Drive where you would like the Job Execution services to be created. This folder should initially, NOT exist (if it is found, the backend will not be deployed) contextName - here you should put the name of the compute context you created in the previous step. dcPath - the physical location on the filesystem to be used for staged data. This is only used at deployment time, it can be configured later in $(appLoc)/services/settings.sas or in the autoexec if used. adminGroup - the name of an existing group, which should have unrestricted access to Data Controller. This is only used at deployment time, it can be configured later in $(appLoc)/services/settings.sas or in the autoexec if used. servertype - should be SASVIYA debug - can stay as false for performance, but could be switched to true for debugging startup issues useComputeApi - use true for best performance. Now, open https://YOURSERVER/DataController (using whichever subfolder you deployed to above) using an account that has the SAS privileges to write to the appLoc location. You will be presented with a deployment screen like the one below. Be sure to check the \"Recreate Database\" option and then click the \"Deploy\" button. Your services are deployed! And the app is operational, albeit still a little sluggish, as every single request is using the APIs to fetch the content of the $(appLoc)/services/settings.sas file. To improve responsiveness by another 700ms we recommend you follow the steps in Update Compute Context Autoexec below.","title":"Deploy frontend"},{"location":"dci-deploysasviya/#deploy-database","text":"If you have a lot of users, such that concurrency (locked datasets) becomes an issue, you might consider migrating the control library to a database. The first part to this is generating the DDL (and inserts). For this, use the DDL exporter as described here . If you need a flavour of DDL that is not yet supported, contact us . Step 2 is simply to run this DDL in your preferred database. Step 3 is to update the library definition in the $(appLoc)/services/settings.sas file using SAS Studio.","title":"Deploy Database"},{"location":"dci-deploysasviya/#update-compute-context-autoexec","text":"First, open the $(appLoc)/services/settings.sas file in SAS Studio, and copy the code. Then, open SASEnvironment Manager, select Contexts, View Compute Contexts, and open the context we created earlier. Switch to the Advanced tab and paste in the SAS code copied from SAS Studio above. It will look similar to: %let DC_LIBREF=DCDBVIYA; %let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}}; %let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}}; libname &dc_libref {{YOUR DC DATABASE}}; To explain each of these lines: DC_LIBREF can be any valid 8 character libref. DC_ADMIN_GROUP is the name of the group which will have unrestricted access to Data Controller DC_STAGING_AREA should point to the location on the filesystem where the staging files and logs are be stored The final libname statement can also be configured to point at a database instead of a BASE engine directory (contact us for DDL) If you have additional libraries that you would like to use in Data Controller, they should also be defined here.","title":"Update Compute Context Autoexec"},{"location":"dci-evaluation/","text":"Data Controller for SAS\u00ae - Evaluation Version \u00b6 Overview \u00b6 A free version of Data Controller is available for evaluation purposes. Compiled into a single SPK, it is very easy to install and configure. However it must not be used in production environments for all the reasons mentioned in the caveats section. Installation \u00b6 Deployment \u00b6 Import \u00b6 Simply import the SPK (using SAS Management Console or Data Integration Studio) to the desired location in the metadata tree. During the import (step 5 of the wizard), be sure to change the location of the library (BASE engine) to a physical directory folder to which the Stored Process system account (eg sassrv ) has write access . Permissions \u00b6 Be sure that the user account you will use in the configuration step below has WRITE METADATA (WM) on the /DataController/services/admin and /DataController/Data folders, and that anyone who will use the app has READ. Configuration \u00b6 Navigate to the web application (eg https://[YOURHOST]/SASStoredProcess?_action=1063 ) and find the location where the app was imported. Then run the DataController/services/admin/configurator stored process. Note Use the same user account as you used to import the SPK, to avoid metadata permissions issues! This may mean logging out / logging back in to the web application. This displays a screen with a choice of SAS Metadata Groups (to which your account belongs) can be chosen. Selecting any of these groups will build / rebuild all the configuration tables (placing logs in a subfolder of the previously configured library location) and provide the chosen group with unrestricted access to the tool. If you do not see any groups, then it is possible your Stored Process is running from a different metadata repository to the location of your SAS users (eg Foundation). To fix this, re-run the configuration stp with the &dc_repo_users=YOURMETAREPO url parameter. Note \"Unrestricted access\" is provided by code logic. Once installed, Data Controller does not ever update or modify metadata. During installation, the services in the /services/admin folder are updated (configuration) or removed (to prevent accidental reinstall). Also the tables are registered in the /Data folder using proc metalib . Usage \u00b6 Simply navigate to the imported location from the Stored Process Web App, right click on the 'clickme' stored process, and open in new window! Caveats \u00b6 The demo version has been optimised for a rapid install, and should not be considered for production / commercial use, or for use by more than 2-5 people, for the following reasons: 1) Static content is compiled into SAS web services, which is inefficient (not scalable) 2) Requires BASE engine for config tables, with high risk of table locks 3) Interface is not licenced for commercial (or production) use, and not supported 4) Underlying macros are not licensed for re-use on other (internal) projects 5) The embedded HandsOnTable library is not licenced for commercial use without a licence key Contact us for a full-featured, fully licenced, scalable and supported deployment of Data Controller at your earliest convenience!","title":"Data Controller for SAS\u00ae - Evaluation Version"},{"location":"dci-evaluation/#data-controller-for-sas-evaluation-version","text":"","title":"Data Controller for SAS\u00ae - Evaluation Version"},{"location":"dci-evaluation/#overview","text":"A free version of Data Controller is available for evaluation purposes. Compiled into a single SPK, it is very easy to install and configure. However it must not be used in production environments for all the reasons mentioned in the caveats section.","title":"Overview"},{"location":"dci-evaluation/#installation","text":"","title":"Installation"},{"location":"dci-evaluation/#deployment","text":"","title":"Deployment"},{"location":"dci-evaluation/#import","text":"Simply import the SPK (using SAS Management Console or Data Integration Studio) to the desired location in the metadata tree. During the import (step 5 of the wizard), be sure to change the location of the library (BASE engine) to a physical directory folder to which the Stored Process system account (eg sassrv ) has write access .","title":"Import"},{"location":"dci-evaluation/#permissions","text":"Be sure that the user account you will use in the configuration step below has WRITE METADATA (WM) on the /DataController/services/admin and /DataController/Data folders, and that anyone who will use the app has READ.","title":"Permissions"},{"location":"dci-evaluation/#configuration","text":"Navigate to the web application (eg https://[YOURHOST]/SASStoredProcess?_action=1063 ) and find the location where the app was imported. Then run the DataController/services/admin/configurator stored process. Note Use the same user account as you used to import the SPK, to avoid metadata permissions issues! This may mean logging out / logging back in to the web application. This displays a screen with a choice of SAS Metadata Groups (to which your account belongs) can be chosen. Selecting any of these groups will build / rebuild all the configuration tables (placing logs in a subfolder of the previously configured library location) and provide the chosen group with unrestricted access to the tool. If you do not see any groups, then it is possible your Stored Process is running from a different metadata repository to the location of your SAS users (eg Foundation). To fix this, re-run the configuration stp with the &dc_repo_users=YOURMETAREPO url parameter. Note \"Unrestricted access\" is provided by code logic. Once installed, Data Controller does not ever update or modify metadata. During installation, the services in the /services/admin folder are updated (configuration) or removed (to prevent accidental reinstall). Also the tables are registered in the /Data folder using proc metalib .","title":"Configuration"},{"location":"dci-evaluation/#usage","text":"Simply navigate to the imported location from the Stored Process Web App, right click on the 'clickme' stored process, and open in new window!","title":"Usage"},{"location":"dci-evaluation/#caveats","text":"The demo version has been optimised for a rapid install, and should not be considered for production / commercial use, or for use by more than 2-5 people, for the following reasons: 1) Static content is compiled into SAS web services, which is inefficient (not scalable) 2) Requires BASE engine for config tables, with high risk of table locks 3) Interface is not licenced for commercial (or production) use, and not supported 4) Underlying macros are not licensed for re-use on other (internal) projects 5) The embedded HandsOnTable library is not licenced for commercial use without a licence key Contact us for a full-featured, fully licenced, scalable and supported deployment of Data Controller at your earliest convenience!","title":"Caveats"},{"location":"dci-requirements/","text":"Data Controller for SAS\u00ae - System Requirements \u00b6 Overview \u00b6 The Data Controller is a SAS Web Application, deployed into an existing SAS platform, and as such has no special requirements beyond what is typically available in a SAS Foundation or Viya environment. SAS 9 \u00b6 Backend \u00b6 A SAS Foundation deployment of at least 9.4M3 must be available. Earlier versions of SAS can be supported, on request. A SAS Stored Process Server must be configured, running under a system account. Mid-Tier \u00b6 A web server with /SASLogon and the SAS SPWA must be available to end users SAS Viya \u00b6 A minimum of Viya 3.5 is recommended to make use of the ability to run a shared compute instance. Frontend \u00b6 All major browsers supported, including IE11 (earlier versions of IE may not work properly). For IE, note that compatibility view must be disabled.","title":"System Requirements"},{"location":"dci-requirements/#data-controller-for-sas-system-requirements","text":"","title":"Data Controller for SAS\u00ae - System Requirements"},{"location":"dci-requirements/#overview","text":"The Data Controller is a SAS Web Application, deployed into an existing SAS platform, and as such has no special requirements beyond what is typically available in a SAS Foundation or Viya environment.","title":"Overview"},{"location":"dci-requirements/#sas-9","text":"","title":"SAS 9"},{"location":"dci-requirements/#backend","text":"A SAS Foundation deployment of at least 9.4M3 must be available. Earlier versions of SAS can be supported, on request. A SAS Stored Process Server must be configured, running under a system account.","title":"Backend"},{"location":"dci-requirements/#mid-tier","text":"A web server with /SASLogon and the SAS SPWA must be available to end users","title":"Mid-Tier"},{"location":"dci-requirements/#sas-viya","text":"A minimum of Viya 3.5 is recommended to make use of the ability to run a shared compute instance.","title":"SAS Viya"},{"location":"dci-requirements/#frontend","text":"All major browsers supported, including IE11 (earlier versions of IE may not work properly). For IE, note that compatibility view must be disabled.","title":"Frontend"},{"location":"dci-stpinstance/","text":"Data Controller for SAS\u00ae - Stored Process Server \u00b6 Overview \u00b6 Data Controller requires that the operating system account (eg sassrv) has the ability to WRITE to each of the libraries set up for editing. For SAS installations where business users have the unrestricted ability to create Stored Processes in production, this can represent a security risk. Under these circumstances, it is recommended to create a dedicated STP server instance for Data Controller, with a dedicated system account. Note Data Controller only updates data (add, delete, modify records). It does not need the ability to create new (permanent) tables, or modify the structure of existing tables. Set up DC account \u00b6 It is recommended to have a user for each environment in which DC is deployed, eg: dcsrv_dev dcsrv_test dcsrv_prod After these OS users are created, log into SMC in relevant environment and open User Manager. Adjust as follows: Open SAS General Servers group Select Accounts tab Add the dcsrv_[ENV] user in DefaultAuth domain STP Server Configuration - 9.4 \u00b6 Open the SAS Deployment Wizard and deploy a new Application Context Server from the panel windows. Be sure to use the relevant dcsrv_[env] user as configured above. Now head to the security section. STP Server Configuration - 9.3 \u00b6 As the wizard does not exist in 9.3 it is necessary to copy the folder structure. Clone existing directory \u00b6 Navigate to the SASApp directory on relevant machine (eg !SASCONFIG/Lev1/SASApp ) and make a copy of the StoredProcessServer folder, and rename it (eg DataControllerSTPsvr). Modify the contents of the new folder as follows: Autoexec (and usermods) \u2013 adjust content to ensure it is relevant to a DC context sasv9_usermods.cfg \u2013 suggested items: - memsize 0 - UTILLOC \u201c/change/only/if/needed\u201d - logconfigloc \"location of DataControllerSTPsvr logconfig.xml file (in new folder)\" The following files should have all instances of \u201c\\StoredProcessServer\\\u201d replaced with \u201c\\DataControllerSTPsvr\\\u201d: Logconfig.xml Logconfig.trace.xml StoredProcessServer.bat Logconfig.apm.xml Sasv9.cfg Dtest folder \u2013 we don\u2019t believe this is used but make the changes anyway (same as above, change all files within it to swap \u201cstoredprocessserver\u201d for DataControllerSTPsvr Sasuser folder \u2013 EMPTY CONTENTS (remove all files). They aren\u2019t relevant in the data controller context. Add Server \u00b6 Open ServerManager and adjust as follows: Log into SMC in relevant environment Open ServerManager Right click / new server Select Application Server Name as \u201cSAS_App_DataController\u201d Click Next / select \u201cStored Process Server\u201d / Next Select \u201cCustom\u201d / Next Command = \u201cC:\\SAS92\\Config\\Lev1\\SASApp\\SASDataEditorStoredProcessServer\\StoredProcessServe r.bat\u201d (adjust as appropriate) Object server parameters = empty Multiuser - select dcsrv_[Env] Choose SASApp server machine (put in RH box) Next / Bridge Connection(default) / Next Bridge Port: 8602 Add / Single Port / 8612 Add / Single Port / 8622 Add / Single Port / 8632 Add at least NINE connections, up to a maximum of (5 per CPU core). Next / finish Next, refresh Server Manager to see the new SAS_App_DataController server. Expand and adjust as follows: Right click SAS_App_DataController-Logical server (first nest), properties, Load Balancing tab, select \u201cResponse Time\u201d Availability timeout \u2013 10 seconds Ok / exit Right click SAS_App_DataController \u2013 Stored Process (second nest), properties, options tab, Advanced options, Load Balancing Max clients 1 Launch timeout \u2013 10 seconds Recycle activation limit \u2013 1 Right click Object Spawner (inside Server Manager) / Properties / Servers, and add the new Data Controller STP from \u201cAvailable Servers\u201d to \u201cSelected Servers\u201d Bounce the object spawner VALIDATION (windows) \u00b6 Open command prompt as an administrator, and run : netstat \u2013aon | find /I \u201c8602\u201d (this will check if the new server is listening on the relevant port) Execute the .bat file to ensure a base sas session can be created in the relevant context ( \u201c!SASConfig\\Lev1\\SASApp\\SASDataEditorStoredProcessServer\\StoredProcessServer.bat\u201d ) In SMC (server manager), right click / validate the new server & test the connection Security \u00b6 STP Server Context \u00b6 To protect the new STP server context, the following initialisation code must be added. This code contains: data _null_; if !('/APPROVED/DC/FOLDER/LOCATION'=:symget('_program')) then do; file _webout; put 'Access to this location has not been approved'; put 'This incident may be reported'; abort cancel; end; run; Save this program in the DataControllerSTPsvr folder. Then open Server Manager in SMC and expand SAS_App_DataController server. Right click SAS_App_DataController-Logical server (first nest), properties, Options tab,Set Server Properties, Request. The init program value should be set to the location of the program above.","title":"SAS 9 Dedicated STP"},{"location":"dci-stpinstance/#data-controller-for-sas-stored-process-server","text":"","title":"Data Controller for SAS\u00ae - Stored Process Server"},{"location":"dci-stpinstance/#overview","text":"Data Controller requires that the operating system account (eg sassrv) has the ability to WRITE to each of the libraries set up for editing. For SAS installations where business users have the unrestricted ability to create Stored Processes in production, this can represent a security risk. Under these circumstances, it is recommended to create a dedicated STP server instance for Data Controller, with a dedicated system account. Note Data Controller only updates data (add, delete, modify records). It does not need the ability to create new (permanent) tables, or modify the structure of existing tables.","title":"Overview"},{"location":"dci-stpinstance/#set-up-dc-account","text":"It is recommended to have a user for each environment in which DC is deployed, eg: dcsrv_dev dcsrv_test dcsrv_prod After these OS users are created, log into SMC in relevant environment and open User Manager. Adjust as follows: Open SAS General Servers group Select Accounts tab Add the dcsrv_[ENV] user in DefaultAuth domain","title":"Set up DC account"},{"location":"dci-stpinstance/#stp-server-configuration-94","text":"Open the SAS Deployment Wizard and deploy a new Application Context Server from the panel windows. Be sure to use the relevant dcsrv_[env] user as configured above. Now head to the security section.","title":"STP Server Configuration - 9.4"},{"location":"dci-stpinstance/#stp-server-configuration-93","text":"As the wizard does not exist in 9.3 it is necessary to copy the folder structure.","title":"STP Server Configuration - 9.3"},{"location":"dci-stpinstance/#clone-existing-directory","text":"Navigate to the SASApp directory on relevant machine (eg !SASCONFIG/Lev1/SASApp ) and make a copy of the StoredProcessServer folder, and rename it (eg DataControllerSTPsvr). Modify the contents of the new folder as follows: Autoexec (and usermods) \u2013 adjust content to ensure it is relevant to a DC context sasv9_usermods.cfg \u2013 suggested items: - memsize 0 - UTILLOC \u201c/change/only/if/needed\u201d - logconfigloc \"location of DataControllerSTPsvr logconfig.xml file (in new folder)\" The following files should have all instances of \u201c\\StoredProcessServer\\\u201d replaced with \u201c\\DataControllerSTPsvr\\\u201d: Logconfig.xml Logconfig.trace.xml StoredProcessServer.bat Logconfig.apm.xml Sasv9.cfg Dtest folder \u2013 we don\u2019t believe this is used but make the changes anyway (same as above, change all files within it to swap \u201cstoredprocessserver\u201d for DataControllerSTPsvr Sasuser folder \u2013 EMPTY CONTENTS (remove all files). They aren\u2019t relevant in the data controller context.","title":"Clone existing directory"},{"location":"dci-stpinstance/#add-server","text":"Open ServerManager and adjust as follows: Log into SMC in relevant environment Open ServerManager Right click / new server Select Application Server Name as \u201cSAS_App_DataController\u201d Click Next / select \u201cStored Process Server\u201d / Next Select \u201cCustom\u201d / Next Command = \u201cC:\\SAS92\\Config\\Lev1\\SASApp\\SASDataEditorStoredProcessServer\\StoredProcessServe r.bat\u201d (adjust as appropriate) Object server parameters = empty Multiuser - select dcsrv_[Env] Choose SASApp server machine (put in RH box) Next / Bridge Connection(default) / Next Bridge Port: 8602 Add / Single Port / 8612 Add / Single Port / 8622 Add / Single Port / 8632 Add at least NINE connections, up to a maximum of (5 per CPU core). Next / finish Next, refresh Server Manager to see the new SAS_App_DataController server. Expand and adjust as follows: Right click SAS_App_DataController-Logical server (first nest), properties, Load Balancing tab, select \u201cResponse Time\u201d Availability timeout \u2013 10 seconds Ok / exit Right click SAS_App_DataController \u2013 Stored Process (second nest), properties, options tab, Advanced options, Load Balancing Max clients 1 Launch timeout \u2013 10 seconds Recycle activation limit \u2013 1 Right click Object Spawner (inside Server Manager) / Properties / Servers, and add the new Data Controller STP from \u201cAvailable Servers\u201d to \u201cSelected Servers\u201d Bounce the object spawner","title":"Add Server"},{"location":"dci-stpinstance/#validation-windows","text":"Open command prompt as an administrator, and run : netstat \u2013aon | find /I \u201c8602\u201d (this will check if the new server is listening on the relevant port) Execute the .bat file to ensure a base sas session can be created in the relevant context ( \u201c!SASConfig\\Lev1\\SASApp\\SASDataEditorStoredProcessServer\\StoredProcessServer.bat\u201d ) In SMC (server manager), right click / validate the new server & test the connection","title":"VALIDATION (windows)"},{"location":"dci-stpinstance/#security","text":"","title":"Security"},{"location":"dci-stpinstance/#stp-server-context","text":"To protect the new STP server context, the following initialisation code must be added. This code contains: data _null_; if !('/APPROVED/DC/FOLDER/LOCATION'=:symget('_program')) then do; file _webout; put 'Access to this location has not been approved'; put 'This incident may be reported'; abort cancel; end; run; Save this program in the DataControllerSTPsvr folder. Then open Server Manager in SMC and expand SAS_App_DataController server. Right click SAS_App_DataController-Logical server (first nest), properties, Options tab,Set Server Properties, Request. The init program value should be set to the location of the program above.","title":"STP Server Context"},{"location":"dci-troubleshooting/","text":"Data Controller for SAS\u00ae - Troubleshooting \u00b6 Overview \u00b6 Let us know if you experience an installation problem that is not described here! Internet Explorer - blank screen \u00b6 If you have an older, or 'locked down' version of Internet Explorer you may get a blank / white screen when navigating to the Data Controller url. To fix this, click settings (cog icon in top right), Compatibility View settings , and uncheck Display intranet sites in Compatibility view as follows: Workspace Server Type Only \u00b6 Data Controller requires the OS account to have disk write privileges for a number of reasons: log capture folder creation (initial setup) table creation (demo version) writing staging data (editors) updating databases / datasets (approvers) On Viya, this is the default case. On SAS 9, if your Stored Process Shared Server account (typically sassrv ) is unavailable, or overly restricted, you may need to use a Workspace Server account for your STPs. This means that your Approvers must have the requisite access to perform the database updates. The imported version of Data Controller is set up to work with the Stored Process Server. To switch this to Workspace Server, you can run the following code after importing the SPK: /* get the macros (or download / %include seperately) */ filename mc url \"https://raw.githubusercontent.com/sasjs/core/main/all.sas\"; %inc mc; /* put the path to your Data Controller folder here */ %let DCROOT=/YOUR/META/PATH/DataController; /* this will extract all the objects in that folder */ %mm_getfoldertree(root=&dcroot, outds=stps) /* this creates the program to update all the STPs in that folder */ filename tmp temp; data _null_; set stps; file tmp; if publictype='StoredProcess' then do; str=cats('%mm_updatestpservertype(target=' ,path,'/',name,',type=WKS)'); put str; end; run; /* run the program */ %inc tmp; Custom Library \u00b6 If you wish to change the default libref or libname then there are TWO items to configure: 1) The library itself 2) The mpelib macro variable and the libname statement in the /Admin/Data_Controller_Settings stored process. Note Be sure to make this change after running the configurator, to ensure the tables are first registered! Permission is needed to access the ServerContext Object \u00b6 After a successful install, your business user may see the following message: Error obtaining stored process from repository Permission is needed to access the ServerContext object attached to the stored process. The reason is that the context chosen when importing the SPK (perhaps, SASApp) is not available to your business user. It's likely you have multiple contexts. The SPK must be re-imported with the correct context chosen. This may require regenerating the tables, or adjusting the permissions, if the new context uses a different system account. Stored Processes Cannot Be Imported Into A Project Repository \u00b6 During the SPK import on a SAS 9 instance you may see the following dialog: Stored processes cannot be imported into a project repository This can happen when importing with Data Integration Studio and your user profile is making use of a personal project repository. Try re-connecting with the Foundation repository, or import with SAS Management Console (which does not support project repositories). There is no LogicalServer of the type requested associated with the ServerContext in metadata. \u00b6 This can happen if you enter the wrong serverName when deploying the SAS program on an EBI platform. Make sure it matches an existing Stored Process Server Context.","title":"Troubleshooting"},{"location":"dci-troubleshooting/#data-controller-for-sas-troubleshooting","text":"","title":"Data Controller for SAS\u00ae - Troubleshooting"},{"location":"dci-troubleshooting/#overview","text":"Let us know if you experience an installation problem that is not described here!","title":"Overview"},{"location":"dci-troubleshooting/#internet-explorer-blank-screen","text":"If you have an older, or 'locked down' version of Internet Explorer you may get a blank / white screen when navigating to the Data Controller url. To fix this, click settings (cog icon in top right), Compatibility View settings , and uncheck Display intranet sites in Compatibility view as follows:","title":"Internet Explorer - blank screen"},{"location":"dci-troubleshooting/#workspace-server-type-only","text":"Data Controller requires the OS account to have disk write privileges for a number of reasons: log capture folder creation (initial setup) table creation (demo version) writing staging data (editors) updating databases / datasets (approvers) On Viya, this is the default case. On SAS 9, if your Stored Process Shared Server account (typically sassrv ) is unavailable, or overly restricted, you may need to use a Workspace Server account for your STPs. This means that your Approvers must have the requisite access to perform the database updates. The imported version of Data Controller is set up to work with the Stored Process Server. To switch this to Workspace Server, you can run the following code after importing the SPK: /* get the macros (or download / %include seperately) */ filename mc url \"https://raw.githubusercontent.com/sasjs/core/main/all.sas\"; %inc mc; /* put the path to your Data Controller folder here */ %let DCROOT=/YOUR/META/PATH/DataController; /* this will extract all the objects in that folder */ %mm_getfoldertree(root=&dcroot, outds=stps) /* this creates the program to update all the STPs in that folder */ filename tmp temp; data _null_; set stps; file tmp; if publictype='StoredProcess' then do; str=cats('%mm_updatestpservertype(target=' ,path,'/',name,',type=WKS)'); put str; end; run; /* run the program */ %inc tmp;","title":"Workspace Server Type Only"},{"location":"dci-troubleshooting/#custom-library","text":"If you wish to change the default libref or libname then there are TWO items to configure: 1) The library itself 2) The mpelib macro variable and the libname statement in the /Admin/Data_Controller_Settings stored process. Note Be sure to make this change after running the configurator, to ensure the tables are first registered!","title":"Custom Library"},{"location":"dci-troubleshooting/#permission-is-needed-to-access-the-servercontext-object","text":"After a successful install, your business user may see the following message: Error obtaining stored process from repository Permission is needed to access the ServerContext object attached to the stored process. The reason is that the context chosen when importing the SPK (perhaps, SASApp) is not available to your business user. It's likely you have multiple contexts. The SPK must be re-imported with the correct context chosen. This may require regenerating the tables, or adjusting the permissions, if the new context uses a different system account.","title":"Permission is needed to access the ServerContext Object"},{"location":"dci-troubleshooting/#stored-processes-cannot-be-imported-into-a-project-repository","text":"During the SPK import on a SAS 9 instance you may see the following dialog: Stored processes cannot be imported into a project repository This can happen when importing with Data Integration Studio and your user profile is making use of a personal project repository. Try re-connecting with the Foundation repository, or import with SAS Management Console (which does not support project repositories).","title":"Stored Processes Cannot Be Imported Into A Project Repository"},{"location":"dci-troubleshooting/#there-is-no-logicalserver-of-the-type-requested-associated-with-the-servercontext-in-metadata","text":"This can happen if you enter the wrong serverName when deploying the SAS program on an EBI platform. Make sure it matches an existing Stored Process Server Context.","title":"There is no LogicalServer of the type requested associated with the ServerContext in metadata."},{"location":"dcu-datacatalog/","text":"Data Controller for SAS: Data Catalog \u00b6 Data Controller collects information about the size and shape of the tables and columns. The Catalog does not contain information about the data content (values). The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables). Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables. The rest is stored on the MPE_DATACATALOG_XXX tables. Tables \u00b6 Libraries \u00b6 This table contains library level attributes to provide a high level overview of data coverage. Note that unless you are an administrator, you are unlikely to have the ability to view / open all of these libraries. To avoid errors when opening invalid libraries, you can add pipe-separated LIBREFs to the DCXXXX.MPE_CONFIG table (var_scope='DC_CATALOG', var_name='DC_IGNORELIBS'). Tables \u00b6 Table attributes are split between those that change infrequently (eg PK_FIELDS) and those that change often (eg size, modified date, and NOBS). Variables \u00b6 Variable attributes come from dictionary tables with an extra PK indicator. A PK is identified by the fact the variable is within an index that is both UNIQUE and NOTNULL. Variable names are always uppercase. Assumptions \u00b6 The following assumptions are made: Data Models (eg attributes) are not sensitive. If so the catalog tables should be disabled. Users can see all tables in the libraries they can access. The refresh process will close out any tables that are not found, if the user can see at least one table in a library. For a particular site, libraries are unique on LIBREF. If you have duplicate librefs, specific table security setups, or sensitive models - contact us.","title":"Data Catalog"},{"location":"dcu-datacatalog/#data-controller-for-sas-data-catalog","text":"Data Controller collects information about the size and shape of the tables and columns. The Catalog does not contain information about the data content (values). The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables). Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables. The rest is stored on the MPE_DATACATALOG_XXX tables.","title":"Data Controller for SAS: Data Catalog"},{"location":"dcu-datacatalog/#tables","text":"","title":"Tables"},{"location":"dcu-datacatalog/#libraries","text":"This table contains library level attributes to provide a high level overview of data coverage. Note that unless you are an administrator, you are unlikely to have the ability to view / open all of these libraries. To avoid errors when opening invalid libraries, you can add pipe-separated LIBREFs to the DCXXXX.MPE_CONFIG table (var_scope='DC_CATALOG', var_name='DC_IGNORELIBS').","title":"Libraries"},{"location":"dcu-datacatalog/#tables_1","text":"Table attributes are split between those that change infrequently (eg PK_FIELDS) and those that change often (eg size, modified date, and NOBS).","title":"Tables"},{"location":"dcu-datacatalog/#variables","text":"Variable attributes come from dictionary tables with an extra PK indicator. A PK is identified by the fact the variable is within an index that is both UNIQUE and NOTNULL. Variable names are always uppercase.","title":"Variables"},{"location":"dcu-datacatalog/#assumptions","text":"The following assumptions are made: Data Models (eg attributes) are not sensitive. If so the catalog tables should be disabled. Users can see all tables in the libraries they can access. The refresh process will close out any tables that are not found, if the user can see at least one table in a library. For a particular site, libraries are unique on LIBREF. If you have duplicate librefs, specific table security setups, or sensitive models - contact us.","title":"Assumptions"},{"location":"dcu-fileupload/","text":"Data Controller for SAS: File Uploads \u00b6 Files can be uploaded via the Editor interface - first choose the library and table, then click \"Upload\". All versions of excel are supported. Uploaded data may optionally contain a column named _____DELETE__THIS__RECORD_____ - where this contains the value \"Yes\" the row is marked for deletion. If loading very large files (eg over 10mb) it is more efficient to use CSV format, as this bypasses the local rendering engine, but also the local DQ checks - so be careful! Examples of local (excel) but not remote (CSV) file checks include: Length of character variables - CSV files are truncated at the max target column length Length of numeric variables - if the target numeric variable is below 8 bytes then the staged CSV value may be rounded if it is too large to fit NOTNULL - this rule is only applied at backend when the constraint is physical (rather than a DC setting) MINVAL MAXVAL CASE Note that the HARDSELECT_*** hooks are not applied to the rendered Excel values (they are currently only applied when editing a cell). Excel Uploads \u00b6 Thanks to our pro license of sheetJS , we can support all versions of excel, large workbooks, and extract data extremely fast. We also support the ingest of password-protected workbooks . The rules for data extraction are: Scan the spreadsheet until a row is found with all the target columns (not case sensitive) Extract data below until the first row containing a blank primary key value This is incredibly flexible, and means: data can be anywhere, on any worksheet data can contain additional columns (they are just ignored) data can be completely surrounded by other data A copy of the original Excel file is also uploaded to the staging area. This means that a complete audit trail can be captured, right back to the original source data. Note If the excel contains more than one range with the target columns (eg, on different sheets), only the FIRST will be extracted. CSV Uploads \u00b6 The following should be considered when uploading data in this way: A header row (with variable names) is required Variable names must match those in the target table (not case sensitive). An easy way to ensure this is to download the data from Viewer and use this as a template. Duplicate variable names are not permitted Missing columns are not permitted Additional columns are ignored The order of variables does not matter EXCEPT for the (optional) _____DELETE__THIS__RECORD_____ variable. When using this variable, it must be the first . The delimiter is extracted from the header row - so for var1;var2;var3 the delimeter would be assumed to be a semicolon The above assumes the delimiter is the first special character! So var,1;var2;var3 would fail The following characters should not be used as delimiters doublequote quote space underscore When loading dates, be aware that Data Controller makes use of the ANYDTDTE and ANYDTDTTME informats (width 19). This means that uploaded date / datetime values should be unambiguous (eg 01FEB1942 vs 01/02/42 ), to avoid confusion - as the latter could be interpreted as 02JAN2042 depending on your locale and options YEARCUTOFF settings. Note that UTC dates with offset values (eg 2018-12-26T09:19:25.123+0100 ) are not currently supported. If this is a feature you would like to see, contact us. Tip To get a copy of a file in the right format for upload, use the file download feature in the Viewer tab Warning Lengths are taken from the target table. If a CSV contains long strings (eg \"ABCDE\" for a $3 variable) then the rest will be silently truncated (only \"ABC\" staged and loaded). If the target variable is a short numeric (eg 4., or 4 bytes) then floats or large integers may be rounded. This issue does not apply to excel uploads, which are first validated in the browser.","title":"File Uploads"},{"location":"dcu-fileupload/#data-controller-for-sas-file-uploads","text":"Files can be uploaded via the Editor interface - first choose the library and table, then click \"Upload\". All versions of excel are supported. Uploaded data may optionally contain a column named _____DELETE__THIS__RECORD_____ - where this contains the value \"Yes\" the row is marked for deletion. If loading very large files (eg over 10mb) it is more efficient to use CSV format, as this bypasses the local rendering engine, but also the local DQ checks - so be careful! Examples of local (excel) but not remote (CSV) file checks include: Length of character variables - CSV files are truncated at the max target column length Length of numeric variables - if the target numeric variable is below 8 bytes then the staged CSV value may be rounded if it is too large to fit NOTNULL - this rule is only applied at backend when the constraint is physical (rather than a DC setting) MINVAL MAXVAL CASE Note that the HARDSELECT_*** hooks are not applied to the rendered Excel values (they are currently only applied when editing a cell).","title":"Data Controller for SAS: File Uploads"},{"location":"dcu-fileupload/#excel-uploads","text":"Thanks to our pro license of sheetJS , we can support all versions of excel, large workbooks, and extract data extremely fast. We also support the ingest of password-protected workbooks . The rules for data extraction are: Scan the spreadsheet until a row is found with all the target columns (not case sensitive) Extract data below until the first row containing a blank primary key value This is incredibly flexible, and means: data can be anywhere, on any worksheet data can contain additional columns (they are just ignored) data can be completely surrounded by other data A copy of the original Excel file is also uploaded to the staging area. This means that a complete audit trail can be captured, right back to the original source data. Note If the excel contains more than one range with the target columns (eg, on different sheets), only the FIRST will be extracted.","title":"Excel Uploads"},{"location":"dcu-fileupload/#csv-uploads","text":"The following should be considered when uploading data in this way: A header row (with variable names) is required Variable names must match those in the target table (not case sensitive). An easy way to ensure this is to download the data from Viewer and use this as a template. Duplicate variable names are not permitted Missing columns are not permitted Additional columns are ignored The order of variables does not matter EXCEPT for the (optional) _____DELETE__THIS__RECORD_____ variable. When using this variable, it must be the first . The delimiter is extracted from the header row - so for var1;var2;var3 the delimeter would be assumed to be a semicolon The above assumes the delimiter is the first special character! So var,1;var2;var3 would fail The following characters should not be used as delimiters doublequote quote space underscore When loading dates, be aware that Data Controller makes use of the ANYDTDTE and ANYDTDTTME informats (width 19). This means that uploaded date / datetime values should be unambiguous (eg 01FEB1942 vs 01/02/42 ), to avoid confusion - as the latter could be interpreted as 02JAN2042 depending on your locale and options YEARCUTOFF settings. Note that UTC dates with offset values (eg 2018-12-26T09:19:25.123+0100 ) are not currently supported. If this is a feature you would like to see, contact us. Tip To get a copy of a file in the right format for upload, use the file download feature in the Viewer tab Warning Lengths are taken from the target table. If a CSV contains long strings (eg \"ABCDE\" for a $3 variable) then the rest will be silently truncated (only \"ABC\" staged and loaded). If the target variable is a short numeric (eg 4., or 4 bytes) then floats or large integers may be rounded. This issue does not apply to excel uploads, which are first validated in the browser.","title":"CSV Uploads"},{"location":"dcu-lineage/","text":"Data Lineage \u00b6 The Data Lineage feature is available for SAS 9 installs. The implementation differs depending on whether the lineage is table level or column level. Table Level lineage \u00b6 Table level lineage is relatively straightforward, and so it is extracted in a single ad-hoc proc metadata call and stored in the DATACTRL.MPE_LINEAGE_TABS table. To trigger the population (or refresh) of this table, simply execute the YOURSERVER/SASStoredProcess/?_program={appLoc}/DataController/admin/refreshtablelineage service from a browser. This data is stored with SCD2 so it is possible to track changes to lineage over time. When users execute table level lineage, queries are made against this table, so there is very little metadata impact. Column Level lineage \u00b6 Column level lineage is more complex as it also includes all the different transforms, and calculation logic along the way. For this reason it is performed at runtime, which means the initial request can take some time if there is a lot of lineage. After the first request, subsequent lineage requests (for that particular column and direction) are cached in the DATACTRL.MPE_LINEAGE_COLS table for faster response times. If the job is changed and a new diagram is needed, the user can click the 'refresh' checkbox. Export Types \u00b6 Both Table and column level lineage pages allow the following export formats: SVG - high res digram format PNG - image format DOT - the graphviz language format used to generate the diagram CSV - a download of all the sources and targets in the diagram","title":"Data Lineage"},{"location":"dcu-lineage/#data-lineage","text":"The Data Lineage feature is available for SAS 9 installs. The implementation differs depending on whether the lineage is table level or column level.","title":"Data Lineage"},{"location":"dcu-lineage/#table-level-lineage","text":"Table level lineage is relatively straightforward, and so it is extracted in a single ad-hoc proc metadata call and stored in the DATACTRL.MPE_LINEAGE_TABS table. To trigger the population (or refresh) of this table, simply execute the YOURSERVER/SASStoredProcess/?_program={appLoc}/DataController/admin/refreshtablelineage service from a browser. This data is stored with SCD2 so it is possible to track changes to lineage over time. When users execute table level lineage, queries are made against this table, so there is very little metadata impact.","title":"Table Level lineage"},{"location":"dcu-lineage/#column-level-lineage","text":"Column level lineage is more complex as it also includes all the different transforms, and calculation logic along the way. For this reason it is performed at runtime, which means the initial request can take some time if there is a lot of lineage. After the first request, subsequent lineage requests (for that particular column and direction) are cached in the DATACTRL.MPE_LINEAGE_COLS table for faster response times. If the job is changed and a new diagram is needed, the user can click the 'refresh' checkbox.","title":"Column Level lineage"},{"location":"dcu-lineage/#export-types","text":"Both Table and column level lineage pages allow the following export formats: SVG - high res digram format PNG - image format DOT - the graphviz language format used to generate the diagram CSV - a download of all the sources and targets in the diagram","title":"Export Types"},{"location":"dcu-tableviewer/","text":"Data Controller for SAS: Viewer \u00b6 The viewer screen provides a raw view of the underlying table. Choose a library, then a table, and click view to see the first 5000 rows. A filter option is provided should you wish to view a different section of rows. The following libraries will be visible: All libraries available on startup (session autoexec) Any libraries configured in the services/public/[Data_Controller_Settings/settings] Stored Process / Viya Job All libraries available to the logged in user in metadata (SAS 9 only) Row and Column level security can also be applied in VIEW mode, as can additional table-level permissions (MPE_SECURITY table). Full Table Search \u00b6 A single search box can be used to make a full table search on any character or numeric value, using this macro . Options \u00b6 This button shows a range of options. If the table is editable, you will also see a EDIT option. Download \u00b6 The Download button gives several options for obtaining the current view of data: 1) CSV. This provides a comma delimited file. 2) Excel. This provides a tab delimited file. 3) SAS Datalines. This provides a SAS program with data as datalines, so that the data can be rebuilt as a SAS table. 4) SAS DDL. A download of a DDL file using SAS flavoured syntax. 5) TSQL DDL. A DDL download using SQL Server flavoured syntax. Note - if the table is registered in Data Controller as being TXTEMPORAL (SCD2) then the download option will prefilter for the current records and removes the valid from / valid to variables. This makes the CSV suitable for DC file upload, if desired. Web Query URL \u00b6 This option gives you a URL that can be used to import data directly into third party tools such as Power BI or Microsoft Excel (as a \"web query\"). You can set up a filter, eg for a particular month, and refresh the query on demand using client tooling such as VBA.","title":"Table Viewer"},{"location":"dcu-tableviewer/#data-controller-for-sas-viewer","text":"The viewer screen provides a raw view of the underlying table. Choose a library, then a table, and click view to see the first 5000 rows. A filter option is provided should you wish to view a different section of rows. The following libraries will be visible: All libraries available on startup (session autoexec) Any libraries configured in the services/public/[Data_Controller_Settings/settings] Stored Process / Viya Job All libraries available to the logged in user in metadata (SAS 9 only) Row and Column level security can also be applied in VIEW mode, as can additional table-level permissions (MPE_SECURITY table).","title":"Data Controller for SAS: Viewer"},{"location":"dcu-tableviewer/#full-table-search","text":"A single search box can be used to make a full table search on any character or numeric value, using this macro .","title":"Full Table Search"},{"location":"dcu-tableviewer/#options","text":"This button shows a range of options. If the table is editable, you will also see a EDIT option.","title":"Options"},{"location":"dcu-tableviewer/#download","text":"The Download button gives several options for obtaining the current view of data: 1) CSV. This provides a comma delimited file. 2) Excel. This provides a tab delimited file. 3) SAS Datalines. This provides a SAS program with data as datalines, so that the data can be rebuilt as a SAS table. 4) SAS DDL. A download of a DDL file using SAS flavoured syntax. 5) TSQL DDL. A DDL download using SQL Server flavoured syntax. Note - if the table is registered in Data Controller as being TXTEMPORAL (SCD2) then the download option will prefilter for the current records and removes the valid from / valid to variables. This makes the CSV suitable for DC file upload, if desired.","title":"Download"},{"location":"dcu-tableviewer/#web-query-url","text":"This option gives you a URL that can be used to import data directly into third party tools such as Power BI or Microsoft Excel (as a \"web query\"). You can set up a filter, eg for a particular month, and refresh the query on demand using client tooling such as VBA.","title":"Web Query URL"},{"location":"dynamic-cell-dropdown/","text":"Dynamic Cell Dropdown \u00b6 This is a simple, but incredibly powerful feature! Configure a SAS process to run when clicking a particular cell. Data Controller will send the row to SAS, and your SAS program can use the values in the row determine a column of values to send back - which will be used in the frontend selectbox. So if you'd like the user to only see products for a particular category, or ISIN's for a particular asset group, you can perform that easily. This feature is used extensively in Data Controller to fetch tables specific to a library, or columns specific to a table: You can also use the response to populate other dropdowns (also in the same row) in the same request - these are called 'extended validations'. Frontend Configuration \u00b6 Open the MPE_VALIDATIONS table and configure the library, table and column that should contain the selectbox. In the RULE_TYPE column, enter either: HARDSELECT_HOOK - The user entry MUST match the returned values SOFTSELECT_HOOK - The user can view the list but type something else if they wish The RULE_VALUE column should contain the full path to the SAS Program, Viya Job or SAS 9 Stored process that you would like to execute. If the value ends in \".sas\" then it is assumed to be a SAS program on a directory, otherwise a SAS web service (STP or Viya Job). Backend Configuration \u00b6 If creating a Stored Process, be sure to deselect the 'automatic SAS macros' - the presence of %stpbegin or %stpend autocall macros will cause problems with the Data Controller backend. You can write any SAS code you wish. For examples of hook scripts you can look at the Data Controller internal validation programs (listed in the MPE_VALIDATIONS table). You will receive the following as inputs: work.source_row -> A dataset containing the current row being modified in Data Controller. This will have already been created in the current SAS session. All variables are available. Use this to filter the initial values in work.dynamic_values . &DC_LIBREF -> The DC control library &LIBDS - The library.dataset being filtered &VARIABLE_NM - The column for which to supply the validation The following tables should be created in the WORK library as outputs: work.dynamic_values work.dynamic_extended_values (optional) WORK.DYNAMIC_VALUES \u00b6 This output table can contain up to three columns: display_index (optional, mandatory if using dynamic_extended_values ). Is a numeric key used to join the two tables. display_value - always character raw_value - unformatted character or numeric according to source data type Example values: DISPLAY_INDEX:best. DISPLAY_VALUE:$ RAW_VALUE 1 $77.43 77.43 2 $88.43 88.43 WORK.DYNAMIC_EXTENDED_VALUES \u00b6 This output table is optional. If provided, it will map the DISPLAY_INDEX from the DYNAMIC_VALUES table to additional column/value pairs, that will be used to populate dropdowns for other cells in the same row. The following columns should be provided: display_index - a numeric key joining each value to the dynamic_values table extra_col_name - the name of the additional variable(s) to contain the extra dropdown(s) display_value - the value to display in the dropdown. Always character. display_type - Either C or N depending on the raw value type raw_value_num - The unformatted value if numeric raw_value_char - The unformatted value if character forced_value - set to 1 to force this value to be automatically selected when the source value is changed. If anything else but 1, the dropdown will still appear, but the user must manually make the selection. Example Values: DISPLAY_INDEX:best. EXTRA_COL_NAME:$32 DISPLAY_VALUE:$ DISPLAY_TYPE:$1. RAW_VALUE_NUM RAW_VALUE_CHAR:$5000 FORCED_VALUE 1 DISCOUNT_RT \"50%\" N 0.5 . 1 DISCOUNT_RT \"40%\" N 0.4 0 1 DISCOUNT_RT \"30%\" N 0.3 1 1 CURRENCY_SYMBOL \"GBP\" C \"GBP\" . 1 CURRENCY_SYMBOL \"RSD\" C \"RSD\" . 2 DISCOUNT_RT \"50%\" N 0.5 . 2 DISCOUNT_RT \"40%\" N 0.4 1 2 CURRENCY_SYMBOL \"EUR\" C \"EUR\" . 2 CURRENCY_SYMBOL \"HKD\" C \"HKD\" 1 Code Examples \u00b6 Simple dropdown /** @file @brief Simple dynamic cell dropdown for product code @details The input table is simply one row from the target table called \"work.source_row\". Available macro variables: @li DC_LIBREF - The DC control library @li LIBDS - The library.dataset being filtered @li VARIABLE_NM - The column being filtered