fix: pipeline

Reviewed-on: #1

.gitea/workflows/publish.yml

@@ -12,20 +12,17 @@ jobs:
       - uses: actions/setup-node@v3
         with:
           node-version: 18
-          
+
       - name: Checkout master
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4  # Updated to latest version
 
       - name: Setup Python
         uses: actions/setup-python@v4
+        with:
+          python-version: '3.12'  # Specify your desired version (e.g., 3.10 or 3.12)
         env:
           AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
           RUNNER_TOOL_CACHE: /opt/hostedtoolcache
-      
-      - name: Install pip3
-        run: |
-          apt-get update
-          apt-get install python3-pip -y
 
       - name: Install Chrome
         run: |
@@ -40,14 +37,13 @@ jobs:
 
       - name: build site
         run: |
-          pip3 install mkdocs
-          pip3 install mkdocs-material
-          pip3 install fontawesome_markdown
-          pip3 install mkdocs-redirects
-          python3 -m mkdocs build --clean
+          pip install mkdocs
+          pip install mkdocs-material
+          pip install fontawesome_markdown
+          pip install mkdocs-redirects
+          mkdocs build --clean
           mkdir site/slides
           npx @marp-team/marp-cli slides/innovation/innovation.md -o ./site/slides/innovation/index.html
           npx @marp-team/marp-cli slides/if/if.md -o site/if.pdf --allow-local-files --html=true
-
       - name: Deploy docs
         run: surfer put --token ${{ secrets.SURFERKEY }} --server docs.datacontroller.io site/* /

.gitpod.yml

@@ -1,8 +0,0 @@
-# This configuration file was automatically generated by Gitpod.
-# Please adjust to your needs (see https://www.gitpod.io/docs/config-gitpod-file)
-# and commit this file to your remote git repository to share the goodness with others.
-
-tasks:
-  - init: npm install
-
-

docs/admin-services.md

@@ -1,98 +1,24 @@
 ---
 layout: article
 title: Admin Services
-description: Data Controller contains a number of admin-only web services, such as DB Export, Lineage Generation, and Data Catalog refresh.
+description: The Administrator Screen provides useful system information and buttons for various administrator actions
+og_title: Administrator Screen
+og_image: /img/admininfo.png
 ---
 
-# Admin Services
+## Administrator Screen
 
-Several web services have been defined to provide additional functionality outside of the user interface.  These somewhat-hidden services must be called directly, using a web browser.  
+The admin screen (under user profile / System) displays a number of useful system parameters as well as several buttons for executing administrator specific actions
 
-In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs).
+![](./img/admininfo.png)
 
-The URL is made up of several components:
+Button info as follows:
 
-* `SERVERURL` -> the domain (and port) on which your SAS server resides
-* `EXECUTOR` -> Either `SASStoredProcess` for SAS 9, else `SASJobExecution` for Viya
-* `APPLOC` -> The root folder location in which the Data Controller backend services were deployed
-* `SERVICE` -> The actual Data Controller service being described.  May include additional parameters.
+|Button|Description|
+|---|---|
+|Refresh Data Catalog|Update Data Catalog for ALL libraries.  More info [here](/dcu-datacatalog).|
+|Download Configuration|This downloads a zip file containing the current database configuration - useful for migrating to a different data controller database instance.|
+|Update Licence Key| Link to the screen for providing a new Data Controller licence key|
+|Export DC Library DDL|COMING SOON!!  <br>Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included|
 
-To illustrate the above, consider the following URL:
 
-[https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL](https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL
-)
-
-This is broken down into:
-
-* `$SERVERURL` = `https://sas.analytium.co.uk`
-* `$EXECUTOR` = `SASJobExecution`
-* `$APPLOC` = `/Public/app/dc`
-* `$SERVICE` = `services/admin/exportdb&flavour=PGSQL`
-
-The below sections will only describe the `$SERVICE` component - you may construct this into a URL as follows:
-
-* `$SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE`
-
-## Export Config
-
-This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance.
-
-EXAMPLE:
-
-* `services/admin/exportconfig`
-
-## Export Database
-Exports the data controller control library in DB specific DDL.  The following URL parameters may be added:
-
-* `&flavour=` (only PGSQL supported at this time)
-* `&schema=` (optional, if target schema is needed)
-
-EXAMPLES:
-
-* `services/admin/exportdb&flavour=PGSQL&schema=DC`
-* `services/admin/exportdb&flavour=PGSQL`
-
-## Refresh Data Catalog
-
-In any SAS estate, it's unlikely the size & shape of data will remain static.  By running a regular Catalog Scan, you can track changes such as:
-
- - Library Properties (size, schema, path, number of tables)
- - Table Properties (size, number of columns, primary keys)
- - Variable Properties (presence in a primary key, constraints, position in the dataset)
-
-The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared?  Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
-
-To run the refresh process, just trigger the stored process, eg below:
-
-* `services/admin/refreshcatalog`
-* `services/admin/refreshcatalog&libref=MYLIB`
-
-The optional `&libref=` parameter allows you to run the process for a single library.  Just provide the libref.
-
-When doing a full scan, the following LIBREFS are ignored:
-
-* 'CASUSER'
-* 'MAPSGFK'
-* 'SASUSER'
-* 'SASWORK
-* 'STPSAMP'
-* 'TEMP'
-* `WORK'
-
-Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them.  This can be useful where there are connection issues for a particular library.
-
-Be aware that the scan process can take a long time if you have a lot of tables!  
-
-Output tables (all SCD2):
-
-* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
-* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
-* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
-* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
-* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)
-
-## Update Licence Key
-
-Whenever navigating Data Controller, there is always a hash (`#`) in the URL.  To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: `/licensing/update`.
-
-If you are using https protocol, you will have 2 keys (licence key / activation key).  In http mode, there is just one key (licence key) for both boxes.

docs/dcc-dates.md

@@ -2,7 +2,8 @@
 
 ## Overview
 
-Dates & datetimes are actually stored as plain numerics in regular SAS tables. In order for the Data Controller to recognise these values as dates / datetimes a format must be applied.
+Dates & datetimes are stored as plain numerics in regular SAS tables. In order for the Data Controller to recognise these values as dates / datetimes a format must be applied.
+
 
 ![displayed](img/dcc-dates1.png) ![source](img/dcc-dates2.png)
 
@@ -14,10 +15,12 @@ Supported date formats:
 * YYMMDD.
 * E8601DA.
 * B8601DA.
+* NLDATE.
 
 Supported datetime formats:
 
 * DATETIME.
+* NLDATM.
 
 Supported time formats:
 
@@ -34,6 +37,9 @@ proc metalib;
 run;
 ```
 
+!!! note
+    Data Controller does not support decimals when EDITING.  For datetimes, this means that values must be rounded to 1 second (milliseconds are not supported).
+    
 If you have other dates / datetimes / times you would like us to support, do [get in touch](https://datacontroller.io/contact)!
 
 

docs/dci-deploysas9.md

@@ -22,15 +22,12 @@ The Stored Processes are deployed using a SAS Program.  This should be executed
 
 ```sas
 %let appLoc=/Shared Data/apps/DataController;  /* CHANGE THIS!! */
+%let serverName=SASApp;
 filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/sas9.sas";
 %inc dc;
 ```
 
-If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), and change the `compiled_apploc` on line 2:  
-
-![](img/sas9_apploc.png)
-
-You can also change the `serverName` here, which is necessary if you are using any other logical server than `SASApp`.
+If you don't have internet access from SAS, download `sas9.sas` from [here](https://git.datacontroller.io/dc/dc/releases), and change the initial `compiled_apploc` and `compiled_serverName` macro variable assignments as necessary.
 
 #### 2 - Deploy the Frontend
 
@@ -43,7 +40,7 @@ Deploy as follows:
 3. Open the `index.html` file and update the values in the `<sasjs>` tag as follows:
 
   * `appLoc` - same as per SAS code in the section above
-  * `serverType` - change this to`SAS9` 
+  * `serverType` - change this to`SAS9`
   * `serverUrl` - Provide only if your SAS Mid Tier is on a different domain than the web server (protocol://SASMIDTIERSERVER:port)
   * `loginMechanism` - set to `Redirected` if using SSO or 2FA
   * `debug` - set to `true` to debug issues on startup (otherwise it's faster to leave it off and turn on in the application itself when needed)
@@ -172,14 +169,15 @@ The full redeployment process is as follows:
     - To a new metadata folder
     - To a new frontend folder (if full deploy)
 * _Delete_ the **new** DC library (metadata + physical tables)
-* _Move_ the **old** DC library (metadata only) to the new DC metadata folder
+* _Move_ the **old** DC library (metadata only) to the new DC metadata folder.  You will need to use DI Studio to do this (you can't move folders using SAS Management Console)
 * Copy the _content_ of the old `services/public/Data_Controller_Settings` STP to the new one
     - This will link the new DC instance to the old DC library / logs directory
     - It will also re-apply any site-specific DC mods
 * Run any/all DB migrations between the old and new DC version
-    - See [migrations](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/db/migrations) folder 
+    - See [migrations](https://git.datacontroller.io/dc/dc/src/branch/main/sas/sasjs/db/migrations) folder
+    - Update the metadata of the SAS Library, using DI Studio, to capture the model changes
 * Test and make sure the new instance works as expected
-* Delete (or rename) the **old** instance 
+* Delete (or rename) the **old** instance
     - Metadata + frontend, NOT the underlying DC library data
 * Rename the new instance so it is the same as the old
     - Both frontend and metadata

docs/dci-troubleshooting.md

@@ -10,6 +10,16 @@ og_image:  https://docs.datacontroller.io/img/cannotimport.png
 ## Overview
 [Let us know](https://datacontroller.io/contact/) if you experience an installation problem that is not described here!
 
+## max number of active processes has been reached for the user
+
+On Viya versions 2025 or later you may get the following message in the network response:
+
+`Unable to create compute server session. Unable to complete the launch request, max number of active processes has been reached for the user: user=USERNAME limit=10`
+
+This limit should be set to at least 20 or 30 due to the way the [sasjs/adapter](https://github.com/sasjs/adapter) works (by prelaunching sessions to improve responsiveness).
+
+A guide for making the configuration change is available [here](https://communities.sas.com/t5/SAS-Communities-Library/Limit-a-user-s-simultaneous-compute-server-processes-in-SAS-Viya/ta-p/761820).
+
 ## Internet Explorer - blank screen
 If you have an older, or 'locked down' version of Internet Explorer you may get a blank / white screen when navigating to the Data Controller url.  To fix this, click settings (cog icon in top right), *Compatibility View settings*, and **uncheck** *Display intranet sites in Compatibility view* as follows:
 ![menu](img/dci-trouble1.png)

docs/dcu-datacatalog.md

@@ -1,7 +1,25 @@
-# Data Controller for SAS: Data Catalog
-Data Controller collects information about the size and shape of the tables and columns.  The Catalog does not contain information about the data content (values).  
+---
+layout: article
+title: DC Data Catalog
+description: Catalog the Libraries, Tables, Columns, SAS Catalogs, and associated Objects in your SAS estate
+og_title: DC Data Catalog Documentation
+og_image: /img/catalog.png
+---
 
-The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
+# DC Data Catalog
+
+In any SAS estate, it's unlikely the size & shape of data will remain static.  By running a regular Catalog Scan, you can track changes such as:
+
+ - Library Properties (size, schema, path, number of tables)
+ - Table Properties (size, number of columns, primary keys)
+ - Variable Properties (presence in a primary key, constraints, position in the dataset)
+ - SAS Catalog Properties (number of entries, created / modified datetimes)
+ - SAS Catalog Object properties (entry name, type, description, created / modified datetimes)
+
+The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared?  Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
+
+
+The Catalog does **not** contain information about the data content (values). It is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
 
 Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables.  The rest is stored on the MPE_DATACATALOG_XXX tables.
 
@@ -20,14 +38,52 @@ Table attributes are split between those that change infrequently (eg PK_FIELDS)
 
 Variable attributes come from dictionary tables with an extra PK indicator.  A PK is identified by the fact the variable is within an index that is both UNIQUE and NOTNULL.  Variable names are always uppercase.
 
+### Catalogs & Objects
+
+This info comes from the dictionary.catalogs table.  The catalog created / modified time is considered to be the earliest created time / latest modified time of the underlying objects.
+
 ## Assumptions
 
 The following assumptions are made:
 
 * Data _Models_ (eg attributes) are not sensitive.  If so the catalog tables should be disabled.
 * Users can see all tables in the libraries they can access.  The refresh process will close out any tables that are not found, if the user can see at least one table in a library.
-* For a particular site, libraries are unique on LIBREF.  
+* For a particular site, libraries are unique on LIBREF.
 
 If you have duplicate librefs, specific table security setups, or sensitive models - contact us.
 
 
+## Refreshing the Data Catalog
+
+The update process for INDIVIDUAL libraries can be run by any user, and is performed in the VIEW menu by expanding a library definition and clicking the refresh icon next to the library name.
+
+![](./img/catalogrefresh.png)
+
+Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page.
+
+When doing a full scan, the following LIBREFS are ignored:
+
+* 'CASUSER'
+* 'MAPSGFK'
+* 'SASUSER'
+* 'SASWORK
+* 'STPSAMP'
+* 'TEMP'
+* `WORK'
+
+Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them.  This can be useful where there are connection issues for a particular library.
+
+Be aware that the scan process can take a long time if you have a lot of tables!
+
+
+Output tables (all SCD2):
+
+* [MPE_DATACATALOG_CATS](/tables/mpe_datacatalog_cats) - SAS Catalog list
+* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
+* [MPE_DATACATALOG_OBJS](/tables/mpe_datacatalog_objs) - SAS Catalog object attributes
+* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
+* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
+* [MPE_DATASTATUS_CATS](/tables/mpe_datastatus_cats) - Frequently changing catalog attributes (such as created / modified datetimes and number of entries)
+* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
+* [MPE_DATASTATUS_OBJS](/tables/mpe_datastatus_objs) - Frequently changing catalog object attributes (such as created / modified datetimes and library concatenation level)
+* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)

docs/deploy-viya.md

@@ -9,7 +9,7 @@ og_image: https://docs.datacontroller.io/img/dci_deploymentdiagramviya.png
 
 ## Overview
 Data Controller for SAS Viya consists of a frontend, a set of Job Execution Services, a staging area, a Compute Context, and a database library.  The library can be a SAS Base engine if desired, however this can cause contention (eg table locks) if end users are able to connect to the datasets directly, eg via Enterprise Guide or Base SAS.
-A database that supports concurrent access is highly recommended.
+A database that supports concurrent access is recommended.
 
 ## Prerequisites
 
@@ -37,29 +37,28 @@ The below areas of the SAS Viya platform are modified when deploying Data Contro
 
 <img src="/img/dci_deploymentdiagramviya.svg" height="350" style="border:3px solid black" >
 
+!!! note
+    The "streaming" version of Viya uses the files API for web content, so there is no need for the web server component.
+
 
 ## Deployment
-Data Controller deployment is split between 3 deployment types:
+Data Controller deployment is split between 2 deployment types:
 
-* Demo version
-* Full Version (manual deploy)
-* Full Version (automated deploy)
+* Streaming (web content served from SAS Drive)
+* Full (web content served from dedicated web server)
 
-<!--
-## Full Version - Manual Deploy
--->
 
 There are several parts to this proces:
 
 1.  Create the Compute Context
-2.  Deploy Frontend
-4.  Prepare the database and update settings (optional)
-5.  Update the Compute Context autoexec
+2.  Deploy Frontend (Full deploy only)
+3.  Deploy Services
+4.  First Run Configuration
+5.  Prepare the database and update settings (optional)
+6.  Update the Compute Context autoexec
 
 
-### Create Compute Context
-
-The Viya Compute context is used to spawn the Job Execution Services - such that those services may run under the specified system account, with a particular autoexec.
+### 1. Create Shared Compute Context
 
 We strongly recommend a dedicated compute context for running Data Controller.  The setup requires an Administrator account.
 
@@ -74,21 +73,60 @@ We strongly recommend a dedicated compute context for running Data Controller.
 
 !!! note
     XCMD is NOT required to use Data Controller.
-### Deploy frontend
+
+### 2. Deploy Frontend (Full Deploy only)
 
 Unzip the frontend into your chosen directory (eg `/var/www/html/DataController`) on the SAS Web Server.  Open `index.html` and update the following inside `dcAdapterSettings`:
 
 - `appLoc` - this should point to the root folder on SAS Drive where you would like the Job Execution services to be created.  This folder should initially, NOT exist (if it is found, the backend will not be deployed)
 - `contextName` - here you should put the name of the compute context you created in the previous step.
-- `dcPath` - the physical location on the filesystem to be used for staged data.  This is only used at deployment time, it can be configured later in `$(appLoc)/services/settings.sas` or in the autoexec if used.
-- `adminGroup` - the name of an existing group, which should have unrestricted access to Data Controller.  This is only used at deployment time, it can be configured later in `$(appLoc)/services/settings.sas` or in the autoexec if used.
 - `servertype` - should be SASVIYA
 - `debug` - can stay as `false` for performance, but could be switched to `true` for debugging startup issues
 - `useComputeApi` - use `true` for best performance.
 
 ![Updating index.html](img/viyadeployindexhtml.png)
 
-Now, open https://YOURSERVER/DataController (using whichever subfolder you deployed to above) using an account that has the SAS privileges to write to the `appLoc` location.
+### 3. Deploy Services
+
+Services are deployed by running a SAS program.
+
+#### Deploy Services: Streaming Deploy
+
+Run the following in SAS Studio:
+
+```sas
+%let apploc=/Public/DataController; /* desired SAS Drive location */
+filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/viya.sas";
+%inc dc;
+```
+
+#### Deploy Services: Full Deploy
+
+Run the following in SAS Studio:
+
+```sas
+%let apploc=/Public/DataController; /* Per configuration in Step 2 above */
+filename dc url "https://git.datacontroller.io/dc/dc/releases/download/latest/viya_noweb.sas";
+%inc dc;
+```
+
+### 4.  First Run Configuration
+
+Now the services are deployed (including the service which creates the staging area) we can open the Data Controller web interface and make the necessary configurations:
+
+#### First Run Configuration: Streaming Deploy
+
+At the end of the SAS log from Step 3, there will be a link (`YOURSAS.SERVER/SASJobExecution?_file=/YOUR/APPLOC/services/DC.html`). Open this to perform the configuration - such as:
+
+* dcpath - physical path for depleyment
+* Admin Group - will have full access to DC
+* Compute Context - should be a shared compute for a multi-user deployment
+
+
+
+#### First Run Configuration: Full Deploy
+
+Open https://YOURSERVER/DataController (using whichever subfolder you deployed to above) using an account that has the SAS privileges to write to the `appLoc` location.
 
 You will be presented with a deployment screen like the one below.  Be sure to check the "Recreate Database" option and then click the "Deploy" button.
 
@@ -96,9 +134,8 @@ You will be presented with a deployment screen like the one below.  Be sure to c
 
 Your services are deployed!  And the app is operational, albeit still a little sluggish, as every single request is using the APIs to fetch the content of the `$(appLoc)/services/settings.sas` file.
 
-To improve responsiveness by another 700ms we recommend you follow the steps in [Update Compute Context Autoexec](/dci-deploysasviya/#update-compute-context-autoexec) below.
 
-### Deploy Database
+### 5. Deploy Database
 If you have a lot of users, such that concurrency (locked datasets) becomes an issue, you might consider migrating the control library to a database.
 
 The first part to this is generating the DDL (and inserts).  For this, use the DDL exporter as described [here](/admin-services/#export-database).  If you need a flavour of DDL that is not yet supported, [contact us](https://datacontroller.io/contact/).
@@ -107,7 +144,9 @@ Step 2 is simply to run this DDL in your preferred database.
 
 Step 3 is to update the library definition in the `$(appLoc)/services/settings.sas` file using SAS Studio.
 
-### Update Compute Context Autoexec
+### 6.Update Compute Context Autoexec
+
+To improve responsiveness by another 700ms we recommend you follow these steps:
 
 First, open the `$(appLoc)/services/settings.sas` file in SAS Studio, and copy the code.
 
@@ -132,54 +171,3 @@ To explain each of these lines:
 * The final libname statement can also be configured to point at a database instead of a BASE engine directory (contact us for DDL)
 
 If you have additional libraries that you would like to use in Data Controller, they should also be defined here.
-<!--
-## Full Version - Automated Deploy
-
-The automated deploy makes use of the SASjs CLI to create the dependent context and job execution services.  In addition to the standard prerequisites (a registered viya system account and a prepared database) you will also need:
-
-* a local copy of the [SASjs CLI](https://sasjs.io/sasjs-cli/#installation)
-* a Client / Secret - with an administrator group in SCOPE, and an authorization_code GRANT_TYPE.  The SASjs [Viya Token Generator](https://github.com/sasjs/viyatoken) may help with this.
-
-### Prepare the Target and Token
-
-To configure this part (one time, manual step), we need to run a single command:
-```
-sasjs add
-```
-A sequence of command line prompts will follow for defining the target.  These prompts are described [here](https://sasjs.io/sasjs-cli-add/).  Note that `appLoc` is the SAS Drive location in which the Data Controller jobs will be deployed.
-
-### Prepare the Context JSON
-This file describes the context that the CI/CD process will generate.  Save this file, eg as `myContext.json`.
-
-```
-{
-  "name": "DataControllerContext",
-  "attributes": {
-    "reuseServerProcesses": true,
-    "runServerAs": "mycasaccount"
-  },
-  "environment": {
-    "autoExecLines": [
-      "%let DC_LIBREF=DCDBVIYA;",
-      "%let DC_ADMIN_GROUP={{YOUR DC ADMIN GROUP}};",
-      "%let DC_STAGING_AREA={{YOUR DEDICATED FILE SYSTEM DRIVE}};",
-      "libname &dc_libref {{YOUR DC DATABASE}};",
-    ],
-    "options": []
-  },
-  "launchContext": {
-    "contextName": "SAS Job Execution launcher context"
-  },
-  "launchType": "service",
-}
-```
-
-### Prepare Deployment Script
-
-The deployment script will run on a build server (or local desktop) and execute as follows:
-
-```
-# Create the SAS Viya Target
-sasjs context create --source myContext.json --target myTarget
-```
--->

docs/roadmap.md

@@ -21,7 +21,6 @@ When features are requested, we will describe the work to be performed in the se
 The following features are currently requested:
 
 * Ability to set 'number of approvals' to zero, enabling instant updates (4 days)
-* Ability to restore previous versions
 * Ability to make automated submissions using an API
 
 
@@ -160,4 +159,15 @@ Our customer was ingesting Basel III reports into SAS and needed an easy to use
 
 We built an approach that allowed end users to define a series of rules for importing cells and ranges from anywhere within a workbook - based on absolute / relative positioning, or using search strings.
 
-The changes we made to deliver this feature are described [here](https://git.datacontroller.io/dc/dc/issues/69) and the final documentation is [here](/excel).
+The changes we made to deliver this feature are described [here](https://git.datacontroller.io/dc/dc/issues/69) and the final documentation is [here](/excel).
+
+
+### Restore Previous Versions
+
+It is now possible to restore any change by heading to the particular staged data screen and hitting the red REVERT button
+
+![](./img/revert.png)
+
+This will submit a NEW change (which must first be approved) that will revert the table to state it was in just prior to the selected upload.
+
+Note that Data Controller can only track (and revert) changes that are made using the Data Controller tool itself! It does not / cannot track changes made externally to a table, by other tools.

docs/tables/mpe_datacatalog_cats.md

@@ -0,0 +1,24 @@
+---
+layout: article
+title: MPE_DATACATALOG_CATS
+description: The MPE_DATACATALOG_CATS table contains all the catalogs available in each library
+og_title: MPE_DATACATALOG_CATS Table Documentation
+og_image: /img/datacatalog_cats.png
+---
+
+# MPE_DATACATALOG_CATS
+
+The `MPE_DATACATALOG_CATS` table contains the catalogs available in each library.
+
+More frequently changing attributes are stored in [MPE_DATASTATUS_CATS](/tables/mpe_datastatus_cats).
+
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
+
+![](/img/datacatalog_cats.png)
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - 🔑 `MEMNAME char(64)`: The catalog member name

docs/tables/mpe_datacatalog_libs.md

@@ -10,11 +10,11 @@ The `MPE_DATACATALOG_LIBS` table catalogs library attributes such as engine, pat
 
 More frequently changing attributes (such as size and number of tables) are stored in [MPE_DATASTATUS_LIBS](/mpe_datastatus_libs).
 
-To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
 
 ## Columns
 
- - `TX_FROM num`: SCD2 open datetime 
+ - `TX_FROM num`: SCD2 open datetime
  - 🔑 `TX_TO num`: SCD2 close datetime
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - `ENGINE char(32)`: The engine used to connect to the library

docs/tables/mpe_datacatalog_objs.md

@@ -0,0 +1,28 @@
+---
+layout: article
+title: MPE_DATACATALOG_OBJS
+description: The MPE_DATACATALOG_OBJS table contains the objects inside every SAS Catalog
+og_title: MPE_DATACATALOG_OBJS Table Documentation
+og_image: /img/datacatalog_objs.png
+---
+
+# MPE_DATACATALOG_OBJS
+
+The `MPE_DATACATALOG_OBJS` table contains a listing of all the objects available in each SAS Catalog.
+
+More frequently changing attributes are stored in [MPE_DATASTATUS_OBJS](/mpe_datastatus_objs).
+
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
+
+![](/img/datacatalog_objs.png)
+
+## Columns
+
+- `TX_FROM num`: SCD2 open datetime
+- 🔑 `TX_TO num`: SCD2 close datetime
+- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+- 🔑 `MEMNAME char(64)`: The catalog member name
+- 🔑 `OBJNAME char(32)`: The object name
+- 🔑 `OBJTYPE char(8)`: The object type
+- `OBJDESC char(256)`: The object description
+- `ALIAS char(32)`: The object alias

docs/tables/mpe_datacatalog_tabs.md

@@ -10,11 +10,11 @@ The `MPE_DATACATALOG_TABS` table catalogs attributes such as number of variables
 
 More frequently changing attributes (such as size modification date and number of observations) are stored in [MPE_DATASTATUS_TABS](/mpe_datastatus_tabs).
 
-To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
 
 ## Columns
 
- - `TX_FROM num`: SCD2 open datetime 
+ - `TX_FROM num`: SCD2 open datetime
  - 🔑 `TX_TO num`: SCD2 close datetime
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - 🔑 `DSN char(64)`: The library member name

docs/tables/mpe_datacatalog_vars.md

@@ -8,11 +8,11 @@ description: The MPE_DATACATALOG_VARS table catalogs variable attributes such as
 
 The `MPE_DATACATALOG_VARS` table catalogs variable attributes such as primary key status, not null constraints and index usage.
 
-To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
 
 ## Columns
 
- - `TX_FROM num`: SCD2 open datetime 
+ - `TX_FROM num`: SCD2 open datetime
  - 🔑 `TX_TO num`: SCD2 close datetime
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - 🔑 `DSN char(64)`: The library member name

docs/tables/mpe_datastatus_cats.md

@@ -0,0 +1,27 @@
+---
+layout: article
+title: MPE_DATASTATUS_CATS
+description: The MPE_DATASTATUS_CATS table captures frequently changing SAS catalog attributes such as created / modified datetimes and number of entries.
+og_title: MPE_DATASTATUS_CATS Table Documentation
+og_image: /img/datastatus_cats.png
+---
+
+# MPE_DATASTATUS_CATS
+
+The `MPE_DATASTATUS_CATS` table captures frequently changing SAS table attributes such as created / modified datetimes and number of entries.
+
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
+
+![](/img/datastatus_cats.png)
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - 🔑 `MEMNAME char(64)`: The catalog member name
+ - `NOBS num`: The number of catalog entries
+ - `CREATED num`: Creation datetime (based on earliest created object)
+ - `MODIFIED num`: Modified datetime (based on last modified object)
+
+

docs/tables/mpe_datastatus_libs.md

@@ -8,11 +8,11 @@ description: The MPE_DATASTATUS_LIBS table captures frequently changing SAS libr
 
 The `MPE_DATASTATUS_LIBS` table captures frequently changing SAS library attributes such as size (if filesystem based) and the number of tables.
 
-To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
 
 ## Columns
 
- - `TX_FROM num`: SCD2 open datetime 
+ - `TX_FROM num`: SCD2 open datetime
  - 🔑 `TX_TO num`: SCD2 close datetime
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - `LIBSIZE num`: The size of the library (in bytes), displayed with the SIZEKMG. format.  Only applicable to BASE engine libraries.

docs/tables/mpe_datastatus_objs.md

@@ -0,0 +1,28 @@
+---
+layout: article
+title: MPE_DATASTATUS_OBJS
+description: The MPE_DATASTATUS_OBJS table captures frequently changing SAS catalog object attributes such as created / modified datetimes and library concatenation level
+og_title: MPE_DATASTATUS_OBJS Table Documentation
+og_image: /img/datastatus_objs.png
+---
+
+# MPE_DATASTATUS_OBJS
+
+The `MPE_DATASTATUS_OBJS` table captures frequently changing SAS catalog object attributes such as created / modified datetimes and library concatenation level
+
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
+
+![](/img/datastatus_objs.png)
+
+## Columns
+
+- `TX_FROM num`: SCD2 open datetime
+- 🔑 `TX_TO num`: SCD2 close datetime
+- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+- 🔑 `MEMNAME char(64)`: The catalog member name
+- 🔑 `OBJNAME char(32)`: The object name
+- 🔑 `OBJTYPE char(8)`: The object type
+- `CREATED num`: Creation datetime (based on earliest created object)
+- `MODIFIED num`: Modified datetime (based on last modified object)
+- `LEVEL num`: Library concatenation level
+

docs/tables/mpe_datastatus_tabs.md

@@ -8,11 +8,11 @@ description: The MPE_DATASTATUS_TABS table captures frequently changing SAS tabl
 
 The `MPE_DATASTATUS_TABS` table captures frequently changing SAS table attributes such as size (if filesystem based), modification date, and the number of observations.
 
-To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).
 
 ## Columns
 
- - `TX_FROM num`: SCD2 open datetime 
+ - `TX_FROM num`: SCD2 open datetime
  - 🔑 `TX_TO num`: SCD2 close datetime
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - 🔑 `DSN char(64)`: The library member name

mkdocs.yml

@@ -18,10 +18,14 @@ nav:
     - MPE_AUDIT: tables/mpe_audit.md
     - MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md
     - MPE_CONFIG: tables/mpe_config.md
+    - MPE_DATACATALOG_CATS: tables/mpe_datacatalog_cats.md
     - MPE_DATACATALOG_LIBS: tables/mpe_datacatalog_libs.md
+    - MPE_DATACATALOG_OBJS: tables/mpe_datacatalog_objs.md
     - MPE_DATACATALOG_TABS: tables/mpe_datacatalog_tabs.md
     - MPE_DATACATALOG_VARS: tables/mpe_datacatalog_vars.md
+    - MPE_DATASTATUS_CATS: tables/mpe_datastatus_cats.md
     - MPE_DATASTATUS_LIBS: tables/mpe_datastatus_libs.md
+    - MPE_DATASTATUS_OBJ: tables/mpe_datastatus_objs.md
     - MPE_DATASTATUS_TABS: tables/mpe_datastatus_tabs.md
     - MPE_LOCKANYTABLE: tables/mpe_lockanytable.md
     - MPE_REQUESTS: tables/mpe_requests.md
@@ -50,7 +54,7 @@ nav:
 - Macros: macros.md
 - Installation:
     - System Requirements: dci-requirements.md
-    - SAS Viya: dci-deploysasviya.md
+    - SAS Viya: deploy-viya.md
     - SAS 9 EBI: dci-deploysas9.md
     - SAS 9 STP Hardening: dci-stpinstance.md
     - Troubleshooting: dci-troubleshooting.md
@@ -105,4 +109,4 @@ theme:
     - JavaScript
     - Bash
 
-copyright: All rights reserved ©2023 Bowe IO Ltd.
+copyright: All rights reserved ©2025 Bowe IO Ltd.