From 198d11a9f2036f2e686bff3e4d99af17d7ac79a9 Mon Sep 17 00:00:00 2001 From: Allan Bowe <> Date: Thu, 16 Apr 2020 22:06:52 +0200 Subject: [PATCH] excel updates --- build.sh | 4 ++-- docs/dci-backend.md | 2 +- docs/dci-frontend.md | 2 +- docs/dcu-fileupload.md | 27 +++++++++++++++++++++++---- docs/licences.md | 4 ++-- 5 files changed, 29 insertions(+), 10 deletions(-) diff --git a/build.sh b/build.sh index aff67a2..cf17f8f 100755 --- a/build.sh +++ b/build.sh @@ -16,12 +16,12 @@ cat > $OUTFILE <<'EOL' ## Overview Data Controller source licences are extracted automatically from our repo using the [license-checker](https://www.npmjs.com/package/license-checker) NPM module - +``` EOL license-checker --production --relativeLicensePath --direct --start ../dcfrontend >> docs/licences.md -echo '' >> docs/licences.md +echo '```' >> docs/licences.md echo 'building mkdocs' mkdocs build --clean diff --git a/docs/dci-backend.md b/docs/dci-backend.md index 8e9241d..b3e01c1 100644 --- a/docs/dci-backend.md +++ b/docs/dci-backend.md @@ -5,7 +5,7 @@ The backend for Data Controller consists of a set of Stored Processes, a macro l ## Regular Deployment -1 - Import `/sas/import.spk` using SAS Management Console. Make a note of the root location in which this was deployed - as this will be added to the `metadataRoot` value in the `h54sConfig.json` file in the [frontend](dci-frontend.md#details) deployment. +1 - Import `/sas/import.spk` using SAS Management Console. Make a note of the root location in which this was deployed - as this will be added to the `metadataRoot` value in the `sasjsConfig.json` file in the [frontend](dci-frontend.md#details) deployment. 2 - Create a physical staging directory. This folder will contain the logs and CSV files generated by Users. The SAS Spawned Server account (eg `sassrv`) will need write access to this location. diff --git a/docs/dci-frontend.md b/docs/dci-frontend.md index ace4bd8..8042469 100644 --- a/docs/dci-frontend.md +++ b/docs/dci-frontend.md @@ -7,7 +7,7 @@ The Data Controller front end comes pre-built, and ready to deploy to the root o 1 - Unzip dcfrontend.zip and upload the entire `datacontroller` directory to the static content server. -2 - Open the `h54s.config` file and update the `metadataRoot` value to the location of the Stored Processes as per [backend](dci-backend.md) deployment. Remember to include the trailing slash (`/`). +2 - Open the `sasjs.config` file and update the `metadataRoot` value to the location of the Stored Processes as per [backend](dci-backend.md) deployment. Remember to include the trailing slash (`/`). It should now be possible to use the application - simply navigate to `YO URSASWEBLOC.domain/yourRoot/datacontroller` and sign in! diff --git a/docs/dcu-fileupload.md b/docs/dcu-fileupload.md index e496747..274b641 100644 --- a/docs/dcu-fileupload.md +++ b/docs/dcu-fileupload.md @@ -1,9 +1,11 @@ # Data Controller for SAS: File Uploads -Files can be uploaded via the Editor interface - first choose the library and table, then click "Upload". Currently only CSV files are supported, although these can be provided with non standard delimiters (such as semicolon). +Files can be uploaded via the Editor interface - first choose the library and table, then click "Upload". All versions of excel are supported. If loading very large files (eg over 10mb) it is more efficient to use CSV format, as this bypasses the local rendering engine, but also the local DQ checks - so be careful! For CSV, alternative delimiters can be used (eg semicolons). +## CSV Uploads + The following should be considered when uploading data in this way: - A header row (with variable names) is required @@ -20,8 +22,25 @@ The following should be considered when uploading data in this way: - space - underscore -When loading dates, be aware that the data controller makes use of the `ANYDTDTE` and `ANYDTDTTME` informats. -This means that uploaded date / datetime values should be unambiguous (eg `01FEB1942` vs `01/02/42`) to avoid confusion - as the latter could be interpreted as `02JAN2042` depending on your locale and options `YEARCUTOFF` settings. +When loading dates, be aware that the data controller makes use of the `ANYDTDTE` and `ANYDTDTTME` informats (width 19). +This means that uploaded date / datetime values should be unambiguous (eg `01FEB1942` vs `01/02/42`) to avoid confusion - as the latter could be interpreted as `02JAN2042` depending on your locale and options `YEARCUTOFF` settings. Note that UTC dates with offset values (eg `2018-12-26T09:19:25.123+0100`) are not currently supported. If this is a feature you would like to see, contact us. !!! tip - To get a copy of a file in the right format for upload, use the [file download](/dc-userguide/#usage) feature in the Viewer tab \ No newline at end of file + To get a copy of a file in the right format for upload, use the [file download](/dc-userguide/#usage) feature in the Viewer tab + +## Excel Uploads + +Thanks to our pro license of [sheetJS](https://sheetjs.com/), we can support all versions of excel, and extract the data super quickly to boot. + +The rules for data extraction are: + +* Scan the spreadsheet until a row is found with all the target columns (with no blank cells between columns) +* Extract data below that row up until the first blank primary key value + +This is incredibly flexible, and means: + +* data can be anywhere, on any worksheet +* data can contain additional columns (they are just ignored) +* data can be completely surrounded by other data + +A copy of the original Excel file is also uploaded to the staging area. This means that a complete audit trail can be captured, right back to the original source data. diff --git a/docs/licences.md b/docs/licences.md index f066a36..2027491 100644 --- a/docs/licences.md +++ b/docs/licences.md @@ -4,7 +4,7 @@ ## Overview Data Controller source licences are extracted automatically from our repo using the [license-checker](https://www.npmjs.com/package/license-checker) NPM module - +``` ├─ @angular/animations@8.2.14 │ ├─ licenses: MIT │ ├─ repository: https://github.com/angular/angular @@ -3829,4 +3829,4 @@ Data Controller source licences are extracted automatically from our repo using ├─ path: /Users/allan/git/dcfrontend/node_modules/zone └─ licenseFile: node_modules/zone/LICENSE.md - +```