diff --git a/.gitpod.yml b/.gitpod.yml deleted file mode 100644 index c4a7726..0000000 --- a/.gitpod.yml +++ /dev/null @@ -1,8 +0,0 @@ -# This configuration file was automatically generated by Gitpod. -# Please adjust to your needs (see https://www.gitpod.io/docs/config-gitpod-file) -# and commit this file to your remote git repository to share the goodness with others. - -tasks: - - init: npm install - - diff --git a/docs/admin-services.md b/docs/admin-services.md index 7ac7bfd..46936a4 100644 --- a/docs/admin-services.md +++ b/docs/admin-services.md @@ -1,98 +1,24 @@ --- layout: article title: Admin Services -description: Data Controller contains a number of admin-only web services, such as DB Export, Lineage Generation, and Data Catalog refresh. +description: The Administrator Screen provides useful system information and buttons for various administrator actions +og_title: Administrator Screen +og_image: /img/admininfo.png --- -# Admin Services +## Administrator Screen -Several web services have been defined to provide additional functionality outside of the user interface. These somewhat-hidden services must be called directly, using a web browser. +The admin screen (under user profile / System) displays a number of useful system parameters as well as several buttons for executing administrator specific actions -In a future version, these features will be made available from an Admin screen (so, no need to manually modify URLs). +![](./img/admininfo.png) -The URL is made up of several components: +Button info as follows: -* `SERVERURL` -> the domain (and port) on which your SAS server resides -* `EXECUTOR` -> Either `SASStoredProcess` for SAS 9, else `SASJobExecution` for Viya -* `APPLOC` -> The root folder location in which the Data Controller backend services were deployed -* `SERVICE` -> The actual Data Controller service being described. May include additional parameters. +|Button|Description| +|---|---| +|Refresh Data Catalog|Update Data Catalog for ALL libraries. More info [here](/dcu-datacatalog).| +|Download Configuration|This downloads a zip file containing the current database configuration - useful for migrating to a different data controller database instance.| +|Update Licence Key| Link to the screen for providing a new Data Controller licence key| +|Export DC Library DDL|COMING SOON!!
Exports the data controller control library in DB specific DDL (eg SAS, PGSQL, TSQL) and allows an optional schema name to be included| -To illustrate the above, consider the following URL: -[https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL](https://viya.4gl.io/SASJobExecution/?_program=/Public/app/viya/services/admin/exportdb&flavour=PGSQL -) - -This is broken down into: - -* `$SERVERURL` = `https://sas.analytium.co.uk` -* `$EXECUTOR` = `SASJobExecution` -* `$APPLOC` = `/Public/app/dc` -* `$SERVICE` = `services/admin/exportdb&flavour=PGSQL` - -The below sections will only describe the `$SERVICE` component - you may construct this into a URL as follows: - -* `$SERVERURL/$EXECUTOR?_program=$APPLOC/$SERVICE` - -## Export Config - -This service will provide a zip file containing the current database configuration. This is useful for migrating to a different data controller database instance. - -EXAMPLE: - -* `services/admin/exportconfig` - -## Export Database -Exports the data controller control library in DB specific DDL. The following URL parameters may be added: - -* `&flavour=` (only PGSQL supported at this time) -* `&schema=` (optional, if target schema is needed) - -EXAMPLES: - -* `services/admin/exportdb&flavour=PGSQL&schema=DC` -* `services/admin/exportdb&flavour=PGSQL` - -## Refresh Data Catalog - -In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as: - - - Library Properties (size, schema, path, number of tables) - - Table Properties (size, number of columns, primary keys) - - Variable Properties (presence in a primary key, constraints, position in the dataset) - -The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs). - -To run the refresh process, just trigger the stored process, eg below: - -* `services/admin/refreshcatalog` -* `services/admin/refreshcatalog&libref=MYLIB` - -The optional `&libref=` parameter allows you to run the process for a single library. Just provide the libref. - -When doing a full scan, the following LIBREFS are ignored: - -* 'CASUSER' -* 'MAPSGFK' -* 'SASUSER' -* 'SASWORK -* 'STPSAMP' -* 'TEMP' -* `WORK' - -Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library. - -Be aware that the scan process can take a long time if you have a lot of tables! - -Output tables (all SCD2): - -* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes -* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes -* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes -* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables) -* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows) - -## Update Licence Key - -Whenever navigating Data Controller, there is always a hash (`#`) in the URL. To access the licence key screen, remove all content to the RIGHT of the hash and add the following string: `/licensing/update`. - -If you are using https protocol, you will have 2 keys (licence key / activation key). In http mode, there is just one key (licence key) for both boxes. diff --git a/docs/dcu-datacatalog.md b/docs/dcu-datacatalog.md index 979dda6..d560ea3 100644 --- a/docs/dcu-datacatalog.md +++ b/docs/dcu-datacatalog.md @@ -1,7 +1,25 @@ -# Data Controller for SAS: Data Catalog -Data Controller collects information about the size and shape of libraries, tables, columns catalogs, and objects. The Catalog does not contain information about the data content (values). +--- +layout: article +title: DC Data Catalog +description: Catalog the Libraries, Tables, Columns, SAS Catalogs, and associated Objects in your SAS estate +og_title: DC Data Catalog Documentation +og_image: /img/catalog.png +--- -The catalog is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables). +# DC Data Catalog + +In any SAS estate, it's unlikely the size & shape of data will remain static. By running a regular Catalog Scan, you can track changes such as: + + - Library Properties (size, schema, path, number of tables) + - Table Properties (size, number of columns, primary keys) + - Variable Properties (presence in a primary key, constraints, position in the dataset) + - SAS Catalog Properties (number of entries, created / modified datetimes) + - SAS Catalog Object properties (entry name, type, description, created / modified datetimes) + +The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared? Just check the history in [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs). + + +The Catalog does **not** contain information about the data content (values). It is based primarily on the existing SAS dictionary tables, augmented with attributes such as primary key fields, filesize / libsize, and number of observations (eg for database tables). Frequently changing data (such as nobs, size) are stored on the MPE_DATASTATUS_XXX tables. The rest is stored on the MPE_DATACATALOG_XXX tables. @@ -35,3 +53,34 @@ The following assumptions are made: If you have duplicate librefs, specific table security setups, or sensitive models - contact us. +## Refreshing the Data Catalog + +The update process for INDIVIDUAL libraries can be run by any user, and is performed in the VIEW menu by expanding a library definition and clicking the refresh icon next to the library name. + +![](./img/catalogrefresh.png) + +Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page. + +When doing a full scan, the following LIBREFS are ignored: + +* 'CASUSER' +* 'MAPSGFK' +* 'SASUSER' +* 'SASWORK +* 'STPSAMP' +* 'TEMP' +* `WORK' + +Additional LIBREFs can be excluded by adding them to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`). Use a pipe (`|`) symbol to seperate them. This can be useful where there are connection issues for a particular library. + +Be aware that the scan process can take a long time if you have a lot of tables! + + +Output tables (all SCD2): + +* [MPE_DATACATALOG_CATS](/tables/mpe_datacatalog_libs) - SAS Catalog list +* [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs) - Library attributes +* [MPE_DATACATALOG_TABS](/tables/mpe_datacatalog_tabs) - Table attributes +* [MPE_DATACATALOG_VARS](/tables/mpe_datacatalog_vars) - Column attributes +* [MPE_DATASTATUS_LIBS](/tables/mpe_datastatus_libs) - Frequently changing library attributes (such as size & number of tables) +* [MPE_DATASTATUS_TABS](/tables/mpe_datastatus_tabs) - Frequently changing table attributes (such as size & number of rows) diff --git a/docs/img/admininfo.png b/docs/img/admininfo.png new file mode 100644 index 0000000..12b2d30 Binary files /dev/null and b/docs/img/admininfo.png differ diff --git a/docs/img/catalog.png b/docs/img/catalog.png new file mode 100644 index 0000000..6373143 Binary files /dev/null and b/docs/img/catalog.png differ diff --git a/docs/img/catalogrefresh.png b/docs/img/catalogrefresh.png new file mode 100644 index 0000000..0393aa1 Binary files /dev/null and b/docs/img/catalogrefresh.png differ diff --git a/docs/tables/mpe_datacatalog_cats.md b/docs/tables/mpe_datacatalog_cats.md new file mode 100644 index 0000000..05e36ab --- /dev/null +++ b/docs/tables/mpe_datacatalog_cats.md @@ -0,0 +1,20 @@ +--- +layout: article +title: MPE_DATACATALOG_CATS +description: The MPE_DATACATALOG_CATS table contains all the catalogs available in each library +--- + +# MPE_DATACATALOG_CATS + +The `MPE_DATACATALOG_CATS` table contains the catalogs available in each library. + +More frequently changing attributes are stored in [MPE_DATASTATUS_CATS](/mpe_datastatus_cats). + +To trigger a scan, see the Refresh Data Catalog [instructions](https://docs.datacontroller.io/admin-services/#refresh-data-catalog). + +## Columns + + - `TX_FROM num`: SCD2 open datetime + - 🔑 `TX_TO num`: SCD2 close datetime + - 🔑 `LIBREF char(8)`: SAS Libref (8 chars) + - 🔑 `MEMNAME char(64)`: The catalog member name diff --git a/mkdocs.yml b/mkdocs.yml index 995c9cf..7c681f7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,6 +18,7 @@ nav: - MPE_AUDIT: tables/mpe_audit.md - MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md - MPE_CONFIG: tables/mpe_config.md + - MPE_DATACATALOG_CATS: tables/mpe_datacatalog_cats.md - MPE_DATACATALOG_LIBS: tables/mpe_datacatalog_libs.md - MPE_DATACATALOG_TABS: tables/mpe_datacatalog_tabs.md - MPE_DATACATALOG_VARS: tables/mpe_datacatalog_vars.md @@ -105,4 +106,4 @@ theme: - JavaScript - Bash -copyright: All rights reserved ©2023 Bowe IO Ltd. +copyright: All rights reserved ©2025 Bowe IO Ltd.