dc/docs.datacontroller.io
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs/roadmap.md
Some checks failed
Publish to docs.datacontroller.io / Deploy docs (push) Has been cancelled
Details

Browse Source
This commit is contained in:
allan 2023-12-22 15:27:32 +00:00
parent aabb86b6f3
commit 197c60615d
1 changed files with 1 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

60
docs/roadmap.md
View File

@ -42,65 +42,7 @@ The following changes are necessary to implement this feature:
### Complex Excel Uploads
When Excel data arrives in multiple ranges, or individual cells, and the cells vary in terms of their column or row identifier, made more "interesting" with the use of merged cells - a rules engine becomes necessary!
This feature enables the use of "EXCEL MAPS". It will enable multiple tables to be loaded in a single import, and that data can be scattered across multiple sheets / cells / ranges, accessed via the rules described further below.
The backend SAS tables must still exist, but the column names do not need to appear in the excel file.
To drive the behaviour, a new configuration table must be added to the control library - MPE_EXCEL_MAP. The columns are defined as follows:
* **XLMAP_ID** - a unique reference for the excel map
* **XLMAP_LIB** - the library of the target table for the data item or range
* **XLMAP_DS** - the target table for the data item or range
* **XLMAP_COL** - the target column for the data item or range
* **XLMAP_SHEET** - the sheet name in which to capture the data. Rules start with a forward slash (/). Example values:
* `Sheet2` - an absolute reference
* `/FIRST` - the first tab in the workbook
* **XLMAP_START** - the rule used to find the top left of the range. Use "R1C1" notation to move the target. Example values:
* `ABSOLUTE F4` - an absolute reference
* `MATCH P R[0]C[2] |My Test` - search column P for the string "My Test" then move 2 columns right
* `MATCH 7 R[-2]C[-1] |Top Banana` - search row 7 for the string "Top Banana" then move 2 rows up and 1 column left
* **XLMAP_FINISH** - The rule used to find the end of the range. Leave blank for individual cells. Example values:
* `BLANKROW` - search down until a blank row is found, then choose the row above it
* `LASTDOWN` - The last non blank cell below the start cell
* `RIGHT 3` - select 3 more cells to the right (4 selected in total)
To illustrate with an example - consider the following excel. The yellow cells need to be imported.
![Complex Excel for SAS Import](/img/excel_map.png)
The data will be imported into two SAS tables - the cells on the left will go into a table with multiple rows, and the cells on the right will be entered as a single row. The XLMAP_ID will also be added to both tables, and the tables will need to have had their keys and quality rules defined in Data Controller in the usual way.
The MPE_EXCEL_MAP configuration entries would be as follows:
|XLMAP_ID|XLMAP_LIB|XLMAP_DS|XLMAP_COL|XLMAP_SHEET|XLMAP_START|XLMAP_FINISH|
|---|---|---|---|---|----|---|
|MAP01|MYLIB|DS1|MI_ITEM|Current Month|`MATCH B R[1]C[0] |ITEM`|`LASTDOWN`|
|MAP01|MYLIB|DS1|MI_AMT|Current Month|`MATCH C R[1]C[0] |AMOUNT`|`LASTDOWN`|
|MAP01|MYLIB|DS2|TMI|Current Month|`ABSOLUTE F6`||
|MAP01|MYLIB|DS2|CB|Current Month|`MATCH F R[2]C[0] |CASH BALANCE`||
|MAP01|MYLIB|DS2|RENT|Current Month|`MATCH E R[0]C[2] |Rent/mortgage`||
|MAP01|MYLIB|DS2|CELL|Current Month|`MATCH E R[0]C[2] |Cell phone`||
To import the excel, the end user simply needs to navigate to the UPLOAD tab, select the appropriate map (eg MAP01), and upload. This will stage two tables (MYLIB.DS1 and MYLIB.DS2) which will go through the usual approval process and quality checks. A copy of the source excel file will be attached to each upload.
#### Estimates
|Component|Estimate (days)|Description|
|---|---|---|
|Frontend|1|Build ExcelMap page with dropdown (and fetching rules), plus drag & drop modal for excel capture|
|Frontend|1|Create staged (unsubmitted) page with support for multiple tables|
|Frontend|2|Create standalone framework utility for rules engine (utils folder), /FIRST sheet rule and Absolute rule|
|Frontend|2|Implement MATCH rule (with BLANKROW, LASTDOWN and {DIRECTION + INTEGER} finish rules)|
|Frontend|0.5|Developer documentation|
|Frontend|2|Cypress tests|
|Backend|0.5|Prep MPE_EXCEL_MAP table, including validations, integrate with DCLIB and add to CI build|
|Backend|1|Create services to fetch Excel Maps and rules (only those the user has permissions for), corresponding SASjs tests, and update developer docs|
|Backend|0.5|Publish online documentation for the overall Excel Maps feature|
Total: **10.5 days**
In Progress - https://git.datacontroller.io/dc/dc/issues/69
### API Submissions