feat: refresh catalog docs

.gitpod.yml

@@ -1,8 +0,0 @@
-# This configuration file was automatically generated by Gitpod.
-# Please adjust to your needs (see https://www.gitpod.io/docs/config-gitpod-file)
-# and commit this file to your remote git repository to share the goodness with others.
-
-tasks:
-  - init: npm install
-
-

docs/admin-services.md

@@ -1,98 +1,24 @@
 ---
 layout: article
 title: Admin Services
-description: Data Controller contains a number of admin-only web services, such as DB Export, Lineage Generation, and Data Catalog refresh.
+description: The Administrator Screen provides useful system information and buttons for various administrator actions
+og_title: Administrator Screen
+og_image: /img/admininfo.png
 ---
 
-# Admin Services
+## Administrator Screen
 
-Several web services have been defined to provide additional functionality outside of the user interface.  These somewhat-hidden services must be called directly, using a web browser.  
+The admin screen (under user profile / System) displays a number of useful system parameters as well as several buttons for executing administrator specific actions
 
-In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs).
+![](./img/admininfo.png)
 
-The URL is made up of several components:
+Button info as follows:
 
-* `SERVERURL` -> the domain (and port) on which your SAS server resides
+|Button|Description|
-* `EXECUTOR` -> Either `SASStoredProcess` for SAS 9, else `SASJobExecution` for Viya
+|---|---|
-* `APPLOC` -> The root folder location in which the Data Controller backend services were deployed
+|Refresh Data Catalog|Update Data Catalog for ALL libraries.  More info [here](/dcu-datacatalog).|
-* `SERVICE` -> The actual Data Controller service being described.  May include additional parameters.
+|Download Configuration|This downloads a zip file containing the current database configuration - useful for migrating to a different data controller database instance.|
+|Update Licence Key| Link to the screen for providing a new Data Controller licence key|
+|Export DC Library DDL|COMING SOON!!  <br>Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included|
 
-To illustrate the above, consider the following URL:
 
-[https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL](https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL
-)
-
-This is broken down into:
-
-* `$SERVERURL` = `https://sas.analytium.co.uk`
-* `$EXECUTOR` = `SASJobExecution`
-* `$APPLOC` = `/Public/app/dc`
-* `$SERVICE` = `services/admin/exportdb&flavour=PGSQL`
-
-The below sections will only describe the `$SERVICE` component - you may construct this into a URL as follows:
-
-* `$SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE`
-
-## Export Config
-
-This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance.
-
-EXAMPLE:
-
-* `services/admin/exportconfig`
-
-## Export Database
-Exports the data controller control library in DB specific DDL.  The following URL parameters may be added:
-
-* `&flavour=` (only PGSQL supported at this time)
-* `&schema=` (optional, if target schema is needed)
-
-EXAMPLES:
-
-* `services/admin/exportdb&flavour=PGSQL&schema=DC`
-* `services/admin/exportdb&flavour=PGSQL`
-
-## Refresh Data Catalog
-
-In any SAS estate, it's unlikely the size & shape of data will remain static.  By running a regular Catalog Scan, you can track changes such as:
-
- - Library Properties (size, schema, path, number of tables)
- - Table Properties (size, number of columns, primary keys)
- - Variable Properties (presence in a primary key, constraints, position in the dataset)
-
-The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared?  Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
-
-To run the refresh process, just trigger the stored process, eg below:
-
-* `services/admin/refreshcatalog`
-* `services/admin/refreshcatalog&libref=MYLIB`
-
-The optional `&libref=` parameter allows you to run the process for a single library.  Just provide the libref.
-
-When doing a full scan, the following LIBREFS are ignored:
-
-* 'CASUSER'
-* 'MAPSGFK'
-* 'SASUSER'
-* 'SASWORK
-* 'STPSAMP'
-* 'TEMP'
-* `WORK'
-
-Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them.  This can be useful where there are connection issues for a particular library.
-
-Be aware that the scan process can take a long time if you have a lot of tables!  
-
-Output tables (all SCD2):
-
-* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
-* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
-* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
-* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
-* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)
-
-## Update Licence Key
-
-Whenever navigating Data Controller, there is always a hash (`#`) in the URL.  To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: `/licensing/update`.
-
-If you are using https protocol, you will have 2 keys (licence key / activation key).  In http mode, there is just one key (licence key) for both boxes.

docs/dcu-datacatalog.md

@@ -1,7 +1,25 @@
-# Data Controller for SAS: Data Catalog
+---
-Data Controller collects information about the size and shape of libraries, tables, columns catalogs, and objects.  The Catalog does not contain information about the data content (values).
+layout: article
+title: DC Data Catalog
+description: Catalog the Libraries, Tables, Columns, SAS Catalogs, and associated Objects in your SAS estate
+og_title: DC Data Catalog Documentation
+og_image: /img/catalog.png
+---
 
-The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
+# DC Data Catalog
+
+In any SAS estate, it's unlikely the size & shape of data will remain static.  By running a regular Catalog Scan, you can track changes such as:
+
+ - Library Properties (size, schema, path, number of tables)
+ - Table Properties (size, number of columns, primary keys)
+ - Variable Properties (presence in a primary key, constraints, position in the dataset)
+ - SAS Catalog Properties (number of entries, created / modified datetimes)
+ - SAS Catalog Object properties (entry name, type, description, created / modified datetimes)
+
+The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared?  Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
+
+
+The Catalog does **not** contain information about the data content (values). It is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
 
 Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables.  The rest is stored on the MPE_DATACATALOG_XXX tables.
 
@@ -35,3 +53,34 @@ The following assumptions are made:
 If you have duplicate librefs, specific table security setups, or sensitive models - contact us.
 
 
+## Refreshing the Data Catalog
+
+The update process for INDIVIDUAL libraries can be run by any user, and is performed in the VIEW menu by expanding a library definition and clicking the refresh icon next to the library name.
+
+![](./img/catalogrefresh.png)
+
+Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page.
+
+When doing a full scan, the following LIBREFS are ignored:
+
+* 'CASUSER'
+* 'MAPSGFK'
+* 'SASUSER'
+* 'SASWORK
+* 'STPSAMP'
+* 'TEMP'
+* `WORK'
+
+Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them.  This can be useful where there are connection issues for a particular library.
+
+Be aware that the scan process can take a long time if you have a lot of tables!
+
+
+Output tables (all SCD2):
+
+* [MPE_DATACATALOG_CATS](/tables/mpe_datacatalog_libs) - SAS Catalog list
+* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
+* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
+* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
+* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
+* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)

docs/tables/mpe_datacatalog_cats.md

@@ -0,0 +1,20 @@
+---
+layout: article
+title: MPE_DATACATALOG_CATS
+description: The MPE_DATACATALOG_CATS table contains all the catalogs available in each library
+---
+
+# MPE_DATACATALOG_CATS
+
+The `MPE_DATACATALOG_CATS` table contains the catalogs available in each library.
+
+More frequently changing attributes are stored in [MPE_DATASTATUS_CATS](/mpe_datastatus_cats).
+
+To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - 🔑 `MEMNAME char(64)`: The catalog member name

mkdocs.yml

@@ -18,6 +18,7 @@ nav:
     - MPE_AUDIT: tables/mpe_audit.md
     - MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md
     - MPE_CONFIG: tables/mpe_config.md
+    - MPE_DATACATALOG_CATS: tables/mpe_datacatalog_cats.md
     - MPE_DATACATALOG_LIBS: tables/mpe_datacatalog_libs.md
     - MPE_DATACATALOG_TABS: tables/mpe_datacatalog_tabs.md
     - MPE_DATACATALOG_VARS: tables/mpe_datacatalog_vars.md
@@ -105,4 +106,4 @@ theme:
     - JavaScript
     - Bash
 
-copyright: All rights reserved ©2023 Bowe IO Ltd.
+copyright: All rights reserved ©2025 Bowe IO Ltd.