dc/docs.datacontroller.io
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: refresh catalog docs
All checks were successful
Publish to docs.datacontroller.io / Deploy docs (push) Successful in 1m31s
Details

Browse Source
This commit is contained in:
allan
2025-07-15 14:45:39 +01:00
parent 4c45779312
commit 2ec7d35342
8 changed files with 87 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
docs/admin-services.md
View File

@@ -1,98 +1,24 @@
---
layout: article
title: Admin Services
description: Data Controller contains a number of admin-only web services, such as DB Export, Lineage Generation, and Data Catalog refresh.
description: The Administrator Screen provides useful system information and buttons for various administrator actions
og_title: Administrator Screen
og_image: /img/admininfo.png
---
# Admin Services
## Administrator Screen
Several web services have been defined to provide additional functionality outside of the user interface. These somewhat-hidden services must be called directly, using a web browser.
The admin screen (under user profile / System) displays a number of useful system parameters as well as several buttons for executing administrator specific actions
In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs).
![](./img/admininfo.png)
The URL is made up of several components:
Button info as follows:
* `SERVERURL` -> the domain (and port) on which your SAS server resides
* `EXECUTOR` -> Either `SASStoredProcess` for SAS 9, else `SASJobExecution` for Viya
* `APPLOC` -> The root folder location in which the Data Controller backend services were deployed
* `SERVICE` -> The actual Data Controller service being described. May include additional parameters.
|Button|Description|
|---|---|
|Refresh Data Catalog|Update Data Catalog for ALL libraries. More info [here](/dcu-datacatalog).|
|Download Configuration|This downloads a zip file containing the current database configuration - useful for migrating to a different data controller database instance.|
|Update Licence Key| Link to the screen for providing a new Data Controller licence key|
|Export DC Library DDL|COMING SOON!! <br>Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included|
To illustrate the above, consider the following URL:
[https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL](https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL
)
This is broken down into:
* `$SERVERURL` = `https://sas.analytium.co.uk`
* `$EXECUTOR` = `SASJobExecution`
* `$APPLOC` = `/Public/app/dc`
* `$SERVICE` = `services/admin/exportdb&flavour=PGSQL`
The below sections will only describe the `$SERVICE` component - you may construct this into a URL as follows:
* `$SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE`
## Export Config
This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance.
EXAMPLE:
* `services/admin/exportconfig`
## Export Database
Exports the data controller control library in DB specific DDL. The following URL parameters may be added:
* `&flavour=` (only PGSQL supported at this time)
* `&schema=` (optional, if target schema is needed)
EXAMPLES:
* `services/admin/exportdb&flavour=PGSQL&schema=DC`
* `services/admin/exportdb&flavour=PGSQL`
## Refresh Data Catalog
In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as:
- Library Properties (size, schema, path, number of tables)
- Table Properties (size, number of columns, primary keys)
- Variable Properties (presence in a primary key, constraints, position in the dataset)
The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
To run the refresh process, just trigger the stored process, eg below:
* `services/admin/refreshcatalog`
* `services/admin/refreshcatalog&libref=MYLIB`
The optional `&libref=` parameter allows you to run the process for a single library. Just provide the libref.
When doing a full scan, the following LIBREFS are ignored:
* 'CASUSER'
* 'MAPSGFK'
* 'SASUSER'
* 'SASWORK
* 'STPSAMP'
* 'TEMP'
* `WORK'
Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library.
Be aware that the scan process can take a long time if you have a lot of tables!
Output tables (all SCD2):
* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)
## Update Licence Key
Whenever navigating Data Controller, there is always a hash (`#`) in the URL. To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: `/licensing/update`.
If you are using https protocol, you will have 2 keys (licence key / activation key). In http mode, there is just one key (licence key) for both boxes.

55
docs/dcu-datacatalog.md
View File

@@ -1,7 +1,25 @@
# Data Controller for SAS: Data Catalog
Data Controller collects information about the size and shape of libraries, tables, columns catalogs, and objects. The Catalog does not contain information about the data content (values).
---
layout: article
title: DC Data Catalog
description: Catalog the Libraries, Tables, Columns, SAS Catalogs, and associated Objects in your SAS estate
og_title: DC Data Catalog Documentation
og_image: /img/catalog.png
---
The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
# DC Data Catalog
In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as:
- Library Properties (size, schema, path, number of tables)
- Table Properties (size, number of columns, primary keys)
- Variable Properties (presence in a primary key, constraints, position in the dataset)
- SAS Catalog Properties (number of entries, created / modified datetimes)
- SAS Catalog Object properties (entry name, type, description, created / modified datetimes)
The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs).
The Catalog does **not** contain information about the data content (values). It is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables).
Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables. The rest is stored on the MPE_DATACATALOG_XXX tables.
@@ -35,3 +53,34 @@ The following assumptions are made:
If you have duplicate librefs, specific table security setups, or sensitive models - contact us.
## Refreshing the Data Catalog
The update process for INDIVIDUAL libraries can be run by any user, and is performed in the VIEW menu by expanding a library definition and clicking the refresh icon next to the library name.
![](./img/catalogrefresh.png)
Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page.
When doing a full scan, the following LIBREFS are ignored:
* 'CASUSER'
* 'MAPSGFK'
* 'SASUSER'
* 'SASWORK
* 'STPSAMP'
* 'TEMP'
* `WORK'
Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library.
Be aware that the scan process can take a long time if you have a lot of tables!
Output tables (all SCD2):
* [MPE_DATACATALOG_CATS](/tables/mpe_datacatalog_libs) - SAS Catalog list
* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes
* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes
* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes
* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables)
* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows)

BIN
docs/img/admininfo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

BIN
docs/img/catalog.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 138 KiB

BIN
docs/img/catalogrefresh.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

20
docs/tables/mpe_datacatalog_cats.md Normal file
View File

@@ -0,0 +1,20 @@
---
layout: article
title: MPE_DATACATALOG_CATS
description: The MPE_DATACATALOG_CATS table contains all the catalogs available in each library
---
# MPE_DATACATALOG_CATS
The `MPE_DATACATALOG_CATS` table contains the catalogs available in each library.
More frequently changing attributes are stored in [MPE_DATASTATUS_CATS](/mpe_datastatus_cats).
To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog).
## Columns
- `TX_FROM num`: SCD2 open datetime
- 🔑 `TX_TO num`: SCD2 close datetime
- 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
- 🔑 `MEMNAME char(64)`: The catalog member name