docs.datacontroller.io/docs/roadmap.md

163 lines
6.9 KiB
Markdown
Raw Normal View History

2021-03-11 22:31:07 +00:00
---
layout: article
title: Roadmap
description: The Data Controller roadmap is aligned with the needs of our customers - we continue to build and prioritise on Features requested by new and existing customers.
2021-03-11 22:31:07 +00:00
og_image: https://i.imgur.com/xFGhgg0.png
---
# Roadmap
## Overview
2021-03-11 22:56:50 +00:00
On this page you can find details of the Features that have currently been requested, that we agree would add value to the product, and are therefore in our development roadmap.
2021-03-11 22:31:07 +00:00
If you would like to see a new Feature added to Data Controller, then let's have a chat!
## Requested Features
When features are requested, we will describe the work to be performed in the sections below.
2021-03-13 10:37:30 +00:00
The following features are currently requested:
2021-03-13 10:37:30 +00:00
2022-04-22 13:15:04 +00:00
* Ability to set 'number of approvals' to zero, enabling instant updates (4 days)
2022-01-27 19:59:42 +00:00
* Ability to restore previous versions
* Ability to make automated submissions using an API
2021-10-09 15:39:29 +00:00
2022-04-22 13:15:04 +00:00
### Set Approvals to Zero
The following changes are necessary to implement this feature:
* Adjust restriction on MPE_TABLES allowing 0 in NUM_APPROVALS_REQD
* Refactor `editors/postdata` enabling execution from `editors/stagedata`
* Additions to the test suite
* Pre-flight checks to ensure the SUBMITTER has APPROVAL rights prior to submit
* Log handling
* Updates to documentation
2022-04-22 13:15:04 +00:00
* Release Notes
2021-10-20 14:27:29 +00:00
2021-10-20 16:22:07 +00:00
### API Submissions
2021-10-19 16:12:16 +00:00
2022-01-17 21:58:32 +00:00
Described [here](https://docs.datacontroller.io/api/).
2021-10-19 15:47:33 +00:00
2021-03-14 18:43:06 +00:00
2021-05-13 11:33:06 +00:00
## Delivered Features
2021-03-14 17:46:22 +00:00
2021-05-13 11:33:06 +00:00
Below are some examples of Features that have been requested (and delivered) into Data Controller.
2021-03-13 10:37:30 +00:00
2022-07-10 15:30:47 +00:00
### Column Level Security
2022-07-20 18:56:46 +00:00
The columns that can be VIEWed or EDITed are now configurable. See [docs](/column-level-security)
2022-07-10 15:30:47 +00:00
### Support for Special SAS Missing Numerics
It is now possible to VIEW and EDIT special SAS missings, eg `.a` through to `.z` and `._`
2022-04-22 13:15:04 +00:00
### Ability to View & Edit Formats
2022-07-20 18:56:46 +00:00
Format Catalogs can now be viewed, filtered, exported and edited. See [docs](/formats)
2022-04-22 13:15:04 +00:00
### Change Tracking Information in a Single Transaction Table
2024-01-24 11:16:43 +00:00
Now available in [mpe_audit](/tables/mpe_audit)
2022-04-22 13:15:04 +00:00
2021-03-11 22:31:07 +00:00
### Dynamic Filtering
2021-05-13 11:33:06 +00:00
Previously, if a user filtered on, say, "region", and then filtered on "store", they would see stores for ALL regions (not just the region/regions already selected in the filter).
2021-03-11 22:31:07 +00:00
![](https://i.imgur.com/KDEVvDi.png)
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-11 22:31:07 +00:00
2021-05-13 11:33:06 +00:00
We added a checkbox to the top left of the filter dialog (default ON) for "Dynamic Where Clause". Whilst enabled, whenever a list of values is requested, it is filtered using every filter clause EXCEPT the one currently being modified.
2021-03-11 22:31:07 +00:00
2021-05-13 11:33:06 +00:00
See [documentation](/filter/). Available from [v.3.12.](https://datacontroller.io/3-12-four-new-data-management-features/)
2021-03-11 22:31:07 +00:00
2021-03-13 10:37:30 +00:00
### Dynamic Cell Validation
2021-03-11 22:31:07 +00:00
2021-05-13 11:33:06 +00:00
When editing a value in a grid, the values presented to the user should be filtered according to additional rules, based on the values of other cells in the same row.
2021-03-11 22:56:00 +00:00
![](https://i.imgur.com/J1q4lqo.png)
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-11 22:56:00 +00:00
2021-05-13 11:33:06 +00:00
We provided two new config item in the MPE_VALIDATIONS table - to links an editable column to a HOOK script via a web service.
2021-03-12 21:33:33 +00:00
The configuration would like like so:
![](https://i.imgur.com/8Hx05GP.png)
2021-03-11 22:56:00 +00:00
2021-05-13 11:33:06 +00:00
In this way, the entire record can be sent to SAS, for processing by the hook script, before returning the desired list of values.
2021-03-11 22:56:00 +00:00
2021-03-12 21:33:33 +00:00
The HOOK_SCRIPT can be either a SAS program on a filesystem (identified by a ".sas" extension) or the path to a registered SAS Service (STP or JES). The latter is identified by the absence of an extension.
2021-03-11 22:56:00 +00:00
This approach provides maximum flexibility for delivering bespoke values in the edit grid dropdown.
2021-05-13 11:33:06 +00:00
See [documentation](/dynamic-cell-dropdown/). Available from [v.3.12.](https://datacontroller.io/3-12-four-new-data-management-features/)
2021-03-14 18:43:06 +00:00
---
2021-03-12 21:33:33 +00:00
2021-03-11 22:56:00 +00:00
### Row Level Security
2021-03-11 22:31:07 +00:00
2021-03-12 21:33:33 +00:00
Row level security is provided by various products in both SAS 9 and Viya, based on the logged in user identity.
This is problematic for the EDIT page, which - by necessity - operates under system account credentials.
It is also the case that some customers need row level security but the data access engine does not support that.
2021-05-13 11:33:06 +00:00
Therefore, there was a need to configure such a feature within the Data Controller product.
2021-03-12 21:33:33 +00:00
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-12 21:33:33 +00:00
2021-05-13 11:33:06 +00:00
A new table (MPE_ROW_LEVEL_SECURITY) was added to the data controller library to allow complex rules to be applied based on the SAS group and the target table. Documentation is [here](/row-level-security/)
2021-03-12 21:33:33 +00:00
2021-05-13 11:33:06 +00:00
Available from [v.3.12.](https://datacontroller.io/3-12-four-new-data-management-features/)
2021-03-12 21:33:33 +00:00
2021-03-13 10:37:30 +00:00
### Formula Preservation
Data Controller uses an OEM licence with the excellent [sheetJS](https://sheetjs.com/) library. This enables us to read pretty much any version of Excel at breakneck speeds.
By default, Data Controller will use the data model of the target table when extracting data, eg to determine whether a column should be character, numeric, date, datetime or time.
2021-05-13 11:33:06 +00:00
Formats used to be ignored and only the cell _values_ would be extracted when formulas are being used.
2021-03-13 10:37:30 +00:00
2021-05-13 11:33:06 +00:00
Now, it is possible to extract and retain the actual formula itself, so it can be re-used when downloading the data again later.
2021-03-13 10:37:30 +00:00
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-13 10:37:30 +00:00
2021-05-13 11:33:06 +00:00
A new table (MPE_EXCEL_CONFIG) was be added to the data controller library to allow the column with the formula to be specified. See [documentation](/excel/)
2021-03-13 10:37:30 +00:00
2021-05-13 11:33:06 +00:00
Available from [v.3.12.](https://datacontroller.io/3-12-four-new-data-management-features/)
2021-03-11 22:56:00 +00:00
2021-03-11 22:31:07 +00:00
### Configurable Locale
When importing spreadsheets with ambiguous dates (eg 01/02 or 02/01) the ANYDTDTM. informat was using the locale of the browser (en_us) instead of that of the client's actual country, resulting in incorrect dates being loaded. This is due to the [default behaviour](https://rawsas.com/look-out-locale-gotcha/) of the SAS Stored Process server.
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-11 22:31:07 +00:00
We added a [new config item](/dcc-options/#dc_locale) so that the locale can be explicitly set for all Data Controller users.
2021-03-12 22:21:55 +00:00
### Restricted Viewer
Data Controller relies on metadata permissions (in SAS 9) or authorization rules (in Viya) to determine who can see which table.
We had a customer who was using Data Controller to provide data access to a company wide audience, most of whom did not have access to SAS client tools (such as Enterprise Guide) and so had not been set up in metadata before.
It was necessary to find a way to restrict the tables which certain groups could see, without having to tweak permissions in SAS Management Console.
2021-10-19 15:47:33 +00:00
**Solution**
2021-03-12 22:21:55 +00:00
2021-10-20 13:48:08 +00:00
We added a [new access level](/dcc-security/#view) in the MPE_SECURITY table so that access could be restricted at both TABLE and LIBRARY level.
2024-01-24 11:51:31 +00:00
### Complex Excel Uploads
Our customer was ingesting Basel III reports into SAS and needed an easy to use mechanism for extracting very specific cells and ranges from a constantly changing workbook format.
**Solution**
We built an approach that allowed end users to define a series of rules for importing cells and ranges from anywhere within a workbook - based on absolute / relative positioning, or using search strings.
The changes we made to deliver this feature are described [here](https://git.datacontroller.io/dc/dc/issues/69) and the final documentation is [here](/excel).