chore(docs): updating data catalog refresh process

docs/dcu-datacatalog.md

@@ -61,6 +61,14 @@ The update process for INDIVIDUAL libraries can be run by any user, and is perfo
 
 Members of the admin group may run the refresh process for ALL libraries by clicking the REFRESH button on the System page.
 
+Under the hood, the refresh is executed by three SAS macros:
+
+- `mpe_refreshlibs` - library attributes (engine, paths, permissions, owners, schemas, metadata name / id) into [MPE_DATACATALOG_LIBS](/tables/mpe_datacatalog_libs)
+- `mpe_refreshtables` - table and column attributes (including primary key detection from constraints and unique not-null indexes) into the DATACATALOG_TABS / VARS tables, and sizes / row counts into the DATASTATUS tables
+- `mpe_refreshcatalogs` - SAS Catalog and object attributes
+
+These run inside the `refreshlibinfo` service (single library, any user) and the `refreshlibs` / `refreshcatalog` services (all libraries, admins only).
+
 When doing a full scan, the following LIBREFS are ignored:
 
 * 'CASUSER'

docs/tables/mpe_datacatalog_libs.md

@@ -25,5 +25,22 @@ To ignore additional librefs, or to trigger a scan, see the Refresh Data Catalog
  - `SCHEMAS char(500)`: The library schema (DB engines)
  - `LIBID char(17)`: The Library Id (from metadata if SAS 9)
 
+## Refresh Process
+
+The table is refreshed by the `mpe_refreshlibs` macro, which runs:
+
+- When a user clicks the refresh icon next to a library in the VIEW menu (via the `refreshlibinfo` service)
+- For ALL libraries when an administrator clicks REFRESH on the System page (via the `refreshlibs` service)
+
+The refresh is driven primarily from `dictionary.libnames`, augmented with library name / id from metadata (SAS 9 only). Noteworthy behaviours:
+
+- The `V9` engine is normalised to `BASE`
+- Concatenated libraries produce multiple quoted entries in `PATHS`, with one comma-separated `PERMS` / `OWNERS` value per path
+- `SCHEMAS` is populated for database engines
+- On SAS 9 (metadata) deployments, invalid libraries are validated by attempting a META libname assignment and skipped on failure. If your environment has invalid libraries that cause exception errors, set the `DC_VIEWLIB_CHECK` config variable to `NO` in Data Controller Settings
+- The following librefs are always excluded: `SASWORK`, `WORK`, `SASUSER`, `CASUSER`, `TEMP`, `STPSAMP`, `MAPSGFK`. Additional librefs can be ignored via `DC_IGNORELIBS` (see [Refreshing the Data Catalog](/dcu-datacatalog/#refreshing-the-data-catalog))
+
+The load is TXTEMPORAL on the `LIBREF` key, so records are closed out (not deleted) when a library disappears.
+
 
 

docs/tables/mpe_datastatus_libs.md

@@ -17,4 +17,14 @@ To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/
  - 🔑 `LIBREF char(8)`: SAS Libref (8 chars)
  - `LIBSIZE num`: The size of the library (in bytes), displayed with the SIZEKMG. format.  Only applicable to BASE engine libraries.
  - `TABLE_CNT num`: The number of tables in the library.
+ - `CATALOG_CNT num`: The number of SAS Catalogs in the library (from `dictionary.catalogs`).
+
+## Refresh Process
+
+This table is populated by the `mpe_refreshtables` macro when a FULL library refresh is performed (all tables). It does not change when a single table is refreshed.
+
+- `LIBSIZE` is the sum of `filesize` across the library members in `dictionary.tables` - hence it is only applicable to BASE (filesystem) engines. For SQL Server libraries, filesize is not available and row counts are instead taken from `sys.partitions` via pass-through
+- `TABLE_CNT` is the count of tables in the library
+
+To trigger a scan, see the Refresh Data Catalog [instructions](/dcu-datacatalog/#refreshing-the-data-catalog).