chore: automated commit

2022-04-30 18:18:08 +01:00 · 2022-04-30 18:18:08 +01:00 · 7caa73e350
commit 7caa73e350
parent aba94a06c7
6 changed files with 127 additions and 7 deletions
--- a/docs/admin-services.md
+++ b/docs/admin-services.md
@ -48,14 +48,40 @@ EXAMPLES:
 * `services/admin/exportdb&flavour=PGSQL&schema=DC`
 * `services/admin/exportdb&flavour=PGSQL`

-## Refresh Catalog
-Refreshes the Data Controller data catalog.  The model is stored in SCD2 so it's a great way to track changes over time!  The process can take a long time if you have a lot of tables - if a library crashes, you can exclude it from the refresh process by adding pipe-separated LIBREFs to the DCXXXX.MPE_CONFIG table (var_scope='DC_CATALOG', var_name='DC_IGNORELIBS').
+## Refresh Data Catalog

-The following params can be added:
+In any SAS estate, it's unlikely the size & shape of data will remain static.  By running a regular Catalog Scan, you can track changes such as:
+ - Library Properties (size, schema, path, number of tables)
+ - Table Properties (size, number of columns, primary keys)
+ - Variable Properties (attributes)

-* `&libref` (optional) to run the process for just one library.
+The data is stored with SCD2 so you can actually **track changes to your model over time**! Curious when that new column appeared?  Just check the history in [MPE_DATACATALOG_TABS](/mpe_datacatalog_tabs.md).

-EXAMPLES:
+To run the refresh process, just trigger the stored process, eg below:

 * `services/admin/refreshcatalog`
-* `services/admin/refreshcatalog&libref=MYLIB`
+* `services/admin/refreshcatalog&libref=MYLIB`
+
+The `&libref=` parameter is optional - if you want to run the process just for a single library.
+
+When doing a full scan, the following LIBREFS are ignored:
+
+* 'CASUSER'
+* 'MAPSGFK'
+* 'SASUSER'
+* 'SASWORK
+* 'STPSAMP'
+* 'TEMP'
+* `WORK'
+
+Be aware that the scan process can take a long time if you have a lot of tables.  Also, note that if a library refresh crashes (due to invalid connection properties), you can exclude it from the subsequent refresh process by adding the `LIBREF` (pipe-separated) to the `DCXXXX.MPE_CONFIG` table (where `var_scope='DC_CATALOG' and var_name='DC_IGNORELIBS'`).
+
+Output tables (all SCD2):
+
+* [MPE_DATACATALOG_LIBS](/mpe_datacatalog_libs.md) - Library attributes
+* [MPE_DATACATALOG_TABS](/mpe_datacatalog_tabs.md) - Table attributes
+* [MPE_DATACATALOG_VARS](/mpe_datacatalog_vars.md) - Column attributes
+* [MPE_DATASTATUS_LIBS](/mpe_datastatus_libs.md) - Frequently changing library attributes (such as size & number of tables)
+* [MPE_DATASTATUS_TABS](/mpe_datastatus_tabs.md) - Frequently changing table attributes (such as size & number of rows)
+
+
--- a/docs/dcu-fileupload.md
+++ b/docs/dcu-fileupload.md
@ -8,7 +8,7 @@ Uploaded data may *optionally* contain a column named `_____DELETE__THIS__RECORD

 ## Excel Uploads

-Thanks to our pro license of [sheetJS](https://sheetjs.com/), we can support all versions of excel, large workbooks, and extract data extremely fast.  We also support the ingest of password-protected workbooks.
+Thanks to our pro license of [sheetJS](https://sheetjs.com/), we can support all versions of excel, large workbooks, and extract data extremely fast.  We also support the ingest of [password-protected workbooks](/videos#Uploading-a-password-protected-excel-file).

 The rules for data extraction are:

--- a/docs/mpe_datacatalog_libs.md
+++ b/docs/mpe_datacatalog_libs.md
@ -0,0 +1,29 @@
+---
+layout: article
+title: MPE_DATACATALOG_LIBS
+description: The MPE_DATACATALOG_LIBS table catalogs library attributes such as engine, paths, permissions, owners & schemas
+---
+
+# MPE_DATACATALOG_LIBS
+
+The `MPE_DATACATALOG_LIBS` table catalogs library attributes such as engine, paths, permissions, owners & schemas.
+
+More frequently changing attributes (such as size and number of tables) are stored in [MPE_DATASTATUS_LIBS](/mpe_datastatus_libs.md).
+
+To ignore additional librefs, or to trigger a scan, see the instructions [here](https://docs.datacontroller.io/admin-services/#refresh-data-catalog)
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime 
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - `ENGINE char(32)`: The engine used to connect to the library
+ - `LIBNAME char(256)`: The Library Name (from metadata if SAS 9)
+ - `PATHS char(8192)`: The directories used (BASE engine only)
+ - `PERMS char(500)`: The directory permissions (BASE engine only)
+ - `OWNERS char(500)`: The directory owners (BASE engine only)
+ - `SCHEMAS char(500)`: The library schema (DB engines)
+ - `LIBID char(17)`: The Library Id (from metadata if SAS 9)
+
+
+
--- a/docs/mpe_datacatalog_tabs.md
+++ b/docs/mpe_datacatalog_tabs.md
@ -0,0 +1,28 @@
+---
+layout: article
+title: MPE_DATACATALOG_TABS
+description: The MPE_DATACATALOG_TABS table catalogs attributes such as number of variables, compression status, and primary key fields.
+---
+
+# MPE_DATACATALOG_TABS
+
+The `MPE_DATACATALOG_TABS` table catalogs attributes such as number of variables, compression status, and primary key fields.
+
+More frequently changing attributes (such as size modification date and number of observations) are stored in [MPE_DATASTATUS_TABS](/mpe_datastatus_tabs.md).
+
+To trigger a scan, see the instructions [here](https://docs.datacontroller.io/admin-services/#refresh-data-catalog)
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime 
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - 🔑 `DSN char(64)`: The library member name
+ - `MEMTYPE char(8)`: The member type
+ - `DBMS_MEMTYPE char(32)`: The DBMS Member Type
+ - `MEMLABEL char(512)`: The Data Set Label
+ - `TYPEMEM char(8)`: The Data Set Type
+ - `NVAR num`: The number of variables
+ - `COMPRESS char(8)`: The compression routine
+ - `PK_FIELDS char(512)`: The list of primary key fields.  These are deduced from the table constraints.  A Primary key column is identified by being in a constraint (or index) that is both UNIQUE and NOT NULL.
+
--- a/docs/mpe_datacatalog_vars.md
+++ b/docs/mpe_datacatalog_vars.md
@ -0,0 +1,31 @@
+---
+layout: article
+title: MPE_DATACATALOG_VARS
+description: The MPE_DATACATALOG_VARS table catalogs variable attributes such as primary key status, not null constraints and index usage.
+---
+
+# MPE_DATACATALOG_TABS
+
+The `MPE_DATACATALOG_VARS` table catalogs variable attributes such as primary key status, not null constraints and index usage.
+
+To trigger a scan, see the instructions [here](https://docs.datacontroller.io/admin-services/#refresh-data-catalog)
+
+## Columns
+
+ - `TX_FROM num`: SCD2 open datetime 
+ - 🔑 `TX_TO num`: SCD2 close datetime
+ - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
+ - 🔑 `DSN char(64)`: The library member name
+ - 🔑 `NAME char(64)`: The variable name
+ - `MEMTYPE char(8)`: The member type
+ - `TYPE char(16)`: The column type
+ - `LENGTH num`: The column length
+ - `VARNUM num`: The column position in the table
+ - `LABEL char(256)`: The column label
+ - `FORMAT char(49)`: The SAS format associated with the column
+ - `IDXUSAGE char(9)`: The column index type
+ - `NOTNULL char(3)`: The NOT NULL status
+ - `PK_IND num`: A flag to say whether the column is part of the primary key (1=PK, 0=Not PK).  A Primary key column is identified by being in a constraint (or index) that is both UNIQUE and NOT NULL.
+
+
+
--- a/docs/videos.md
+++ b/docs/videos.md
@ -95,3 +95,9 @@ In unix environments, when attempting to edit a locked table, users are notified

 <iframe src="https://player.vimeo.com/video/293557084" width="640" height="360" frameborder="0" allowfullscreen></iframe>

+## Uploading a Password Protected Excel File
+
+With Data Controller you can upload password protected excel files into SAS.  These are opened in the browser (thanks to our OEM licence of SheetJS) and the underlying data is extracted automatically.
+
+<iframe width="560" height="315" src="https://www.youtube.com/watch?v=b2lfBCkdvBI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+