docs(developer-guide): add a Developer Guide section
Two pages, both aimed at SAS developers rather than end users: - Filter in a Pre-Edit Hook: what editors/getdata returns (sasparams.FILTER_TEXT and the query clauses, never FILTER_RK), when PRE_EDIT_HOOK runs relative to the filter, and a worked example that widens a date filter to the previous quarter - re-reading the table so the rows match, and keeping Row Level Security applied to both branches. - One Table, Two Rule Sets: moved from the datacontroller.io blog, so the developer-facing material lives in one place. Images moved to docs/img/.
This commit is contained in:
dc committed
1 parent
c5d6806d53
commit
cc91f8b1f9
7 files changed
+266
No files matched your search
@@ -0,0 +1,141 @@
|
||||
---
|
||||
layout: article
|
||||
title: Updating the Filter in a Pre-Edit Hook
|
||||
description: Read the filter a user applied, and hand back a modified one, from a Data Controller pre-edit hook - including changing the rows the editor shows.
|
||||
---
|
||||
|
||||
# Updating the Filter in a Pre-Edit Hook
|
||||
|
||||
A [pre-edit hook](/dcc-tables/#pre_edit_hook) can read the filter a user applied and return a modified one. This is useful when the editor should widen a user's selection behind the scenes - for example, when a user filters for the current quarter and the grid should also show the previous one.
|
||||
|
||||
The hook runs inside the `editors/getdata` service, so both the applied filter and the filtered rows are in scope. This page describes what is available at that point and works through an example.
|
||||
|
||||
## What the editor receives
|
||||
|
||||
When a filtered editor loads, `editors/getdata` returns the filter to the client in two forms:
|
||||
|
||||
- `sasparams.FILTER_TEXT` - the resolved WHERE clause, which the editor shows above the grid.
|
||||
- `query` - the filter's clauses as structured rows (`variable_nm`, `operator_nm`, `raw_value`), which the editor renders as the filter chips.
|
||||
|
||||
The response does not carry the filter key (`FILTER_RK`). That is an *input* to `getdata`, not an output - the editor already holds it in the URL (`/editor/<table>/<FILTER_RK>`), so there is nothing to return.
|
||||
|
||||
## When the hook runs
|
||||
|
||||
`getdata` builds the filter, applies it, and sorts the result, all before the pre-edit hook runs. In order:
|
||||
|
||||
1. the WHERE clause is written to a temporary fileref, `filtref`;
|
||||
2. `work.OUT` is populated with `where %inc filtref` - the filter is applied here;
|
||||
3. the pre-edit hook runs;
|
||||
4. `FILTER_TEXT` is read back out of `filtref`, and the response is streamed.
|
||||
|
||||
So the hook sits between "the rows are filtered" and "the filter is returned". Two consequences follow:
|
||||
|
||||
- It can change what the editor is told the filter was, by rewriting `filtref` and, if the chips must match, `work.query`.
|
||||
- It cannot change which rows came back *that* way, because the data step has already run. To return different rows, the hook must re-read the table itself.
|
||||
|
||||
## Take the value from the data
|
||||
|
||||
A hook that wants to extend a date filter needs the date the user is working with. It does not have to parse the filter text for it - the filtered rows are already in `work.OUT`, so the value can be read from there.
|
||||
|
||||
## Example: widen a date filter to the previous quarter
|
||||
|
||||
The hook below is registered in `MPE_TABLES.PRE_EDIT_HOOK` for a table with a date column `ORDER_DATE`. When a user opens the editor with a filter that includes `ORDER_DATE`, it extends the filter to also cover the previous quarter, re-reads the table so the grid shows both quarters, and returns the widened filter text.
|
||||
|
||||
```sas
|
||||
/* PRE_EDIT_HOOK: widen the user's ORDER_DATE filter by one quarter */
|
||||
|
||||
%if &filter_rk > 0 %then %do;
|
||||
|
||||
/* the date the user is looking at, taken from the rows already returned */
|
||||
%let refdt=;
|
||||
proc sql noprint;
|
||||
select max(ORDER_DATE) into :refdt from work.out;
|
||||
quit;
|
||||
|
||||
%if &refdt ne %then %do;
|
||||
|
||||
/* the previous quarter, relative to that date */
|
||||
%let pq_from=%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,b)),date9.));
|
||||
%let pq_to =%sysfunc(putn(%sysfunc(intnx(quarter,&refdt,-1,e)),date9.));
|
||||
|
||||
/* the user's own clauses, as a standalone WHERE */
|
||||
%mp_filtergenerate(work.query,outref=userref)
|
||||
|
||||
/* Data Controller's own clause - current-record validity, plus RLS */
|
||||
%mpe_filtermaster(EDIT,&orig_libds,
|
||||
dclib=&mpelib,
|
||||
filter_rk=-1,
|
||||
outref=ctlref,
|
||||
outds=work.ctlq
|
||||
)
|
||||
|
||||
/* read each clause (which may span several lines) into a macro variable */
|
||||
%let usertext=;
|
||||
data _null_;
|
||||
infile userref end=eof;
|
||||
input;
|
||||
length t $32767;
|
||||
retain t;
|
||||
t=catx(' ',t,_infile_);
|
||||
if eof then call symputx('usertext',t);
|
||||
run;
|
||||
%let ctltext=;
|
||||
data _null_;
|
||||
infile ctlref end=eof;
|
||||
input;
|
||||
length t $32767;
|
||||
retain t;
|
||||
t=catx(' ',t,_infile_);
|
||||
if eof then call symputx('ctltext',t);
|
||||
run;
|
||||
|
||||
/* ( user's filter OR previous quarter ) AND ( validity + RLS ) */
|
||||
filename newfilt temp;
|
||||
data _null_;
|
||||
file newfilt lrecl=32767;
|
||||
length f $32767;
|
||||
f='(( '!!trim(symget('usertext'))!!' ) OR (ORDER_DATE between "&pq_from"d and "&pq_to"d))'
|
||||
!!' AND ('!!trim(symget('ctltext'))!!' )';
|
||||
put f;
|
||||
run;
|
||||
|
||||
/* re-read the table, so the grid shows both quarters */
|
||||
data work.out (drop=&var_txfrom &var_txto &var_processed);
|
||||
_____DELETE__THIS__RECORD_____="No";
|
||||
set &orig_libds;
|
||||
where %inc newfilt;
|
||||
run;
|
||||
|
||||
/* hand the widened filter back to the editor */
|
||||
data _null_;
|
||||
file filtref lrecl=32767;
|
||||
infile newfilt;
|
||||
input;
|
||||
put _infile_;
|
||||
run;
|
||||
|
||||
%end;
|
||||
%end;
|
||||
```
|
||||
|
||||
### What the hook does
|
||||
|
||||
- **The reference date.** `work.OUT` holds the rows the user's filter returned, so the latest `ORDER_DATE` on screen is read straight from it. Nothing has to be parsed out of the filter text.
|
||||
- **The previous quarter.** `INTNX` moves back one quarter from that date and returns its first and last day, formatted as SAS date literals (`01APR2026` and `30JUN2026`, for example).
|
||||
- **The user's clause.** `work.QUERY` is the filter's clauses as structured rows, and `%mp_filtergenerate` turns them back into a WHERE clause. This is the user's filter on its own, with no Row Level Security or validity clause mixed in.
|
||||
- **Data Controller's clause.** Calling `%mpe_filtermaster` with `filter_rk=-1` produces the part of the filter that is not the user's: the current-record validity clause and any Row Level Security, or `1=1` when there is neither.
|
||||
- **The combined filter.** The two clauses are combined as `( user OR previous-quarter ) AND ( validity + RLS )`, so the previous quarter is added to the user's selection without widening anything the user is not entitled to see.
|
||||
- **The rows.** `work.OUT` is rebuilt with the combined filter, mirroring the data step `getdata` uses itself - including dropping the transaction and processing columns it drops.
|
||||
- **The returned text.** `filtref` is overwritten with the combined filter, so the clause the editor shows is the one that was actually applied.
|
||||
|
||||
## Row Level Security
|
||||
|
||||
The user's clause and Data Controller's clause have to be kept apart, because Row Level Security lives in the second one. Combining them as `( user OR previous-quarter ) AND ( validity + RLS )` keeps RLS applied to both branches. Wrapping the whole existing filter instead - `( existing OR previous-quarter )` - would let the previous-quarter rows through without their RLS condition, so do not take that shortcut.
|
||||
|
||||
## The row cap
|
||||
|
||||
`getdata` refuses a result larger than `DC_MAXOBS_WEBEDIT` (500 by default) with *Table is too big*. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`.
|
||||
|
||||
## Matching the filter chips
|
||||
|
||||
The clause above the grid comes from `FILTER_TEXT`, so overwriting `filtref` is what updates it. The filter chips are rendered from the `query` output instead, so if they must reflect the widened filter too, amend `work.QUERY` in the same hook - append the extra clause as a new subgroup with `group_logic='OR'` before the response is built.
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
layout: article
|
||||
title: One Table, Two Rule Sets
|
||||
description: A single physical table can carry only one set of Data Controller validation rules. Two librefs over the same data - or a pair of hook scripts - give you as many rule sets as you need.
|
||||
---
|
||||
|
||||
# One Table, Two Rule Sets
|
||||
|
||||
Most Data Controller sites settle into an obvious mapping: one table, one edit screen, one set of validation rules. But that is not always what the business wants. A table of orders might be edited from a finance report that only tolerates small adjustments, and from an operations report where much larger ones are routine. Same table, same columns, same approvers - different rules.
|
||||
|
||||
Data Controller's validation rules are configured per table, so this takes a little thought. There are two ways to do it: the one we recommend, and the one to reach for when the first is not available.
|
||||
|
||||
## Why one table is one rule set
|
||||
|
||||
Two configuration tables decide this.
|
||||
|
||||
`MPE_TABLES` is the list of editable tables, and its primary key is `(tx_from, libref, dsn)`. One physical table is one editable table.
|
||||
|
||||
`MPE_VALIDATIONS` holds the rules, and its primary key is `(tx_from, base_lib, base_ds, base_col, rule_type)`. Rules hang off a physical `libref.dataset`. There is no per-menu or per-report scoping anywhere in the schema, and the editor is handed exactly the rules whose `base_lib` and `base_ds` match the table being opened.
|
||||
|
||||
So two rule sets on one table need two distinct `libref.dataset` identities. The question is how to get them without copying the data.
|
||||
|
||||
## Option 1 (recommended): two librefs over the same data
|
||||
|
||||
A libref is just a name pointing at a location. Nothing stops you assigning two of them to the same place, and Data Controller will treat the two as separate tables:
|
||||
|
||||
```sas
|
||||
libname ORDERS_EU '/data/orders';
|
||||
libname ORDERS_US '/data/orders';
|
||||
```
|
||||
|
||||
`ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` are now the same physical file, but they are different rows in `MPE_TABLES` and can therefore carry different rows in `MPE_VALIDATIONS`. Register both, give each its own rules, and point each report at its own editor URL - `#/editor/ORDERS_EU.ORDERS` and `#/editor/ORDERS_US.ORDERS`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Everything else works exactly as it always did. Filtering, search, the row cap, the approval diff and the audit trail all operate on the table as normal, because as far as Data Controller is concerned these are ordinary tables. The only difference is that the two names resolve to the same file, so an approval in either report updates the same data - and that one difference has a consequence for concurrency, which is covered below.
|
||||
|
||||
There is no copy to keep in sync, no hook to write and nothing to maintain. That is why it is the option we recommend.
|
||||
|
||||
### Things worth knowing
|
||||
|
||||
- Where the librefs are defined depends on your platform. On Viya, in the compute context's `autoexec.sas` (or `[DC Drive Path]/services/settings.sas`); on SAS 9, as metadata libraries or in the Data Controller Settings stored process; on SASjs Server, in `services/public/settings.sas`. The one requirement is that each library has a unique libref.
|
||||
- `mp_lockanytable` keys on `libref.dataset`, so the two menus do not serialise against each other. That has consequences beyond a collision - see *The concurrency caveat* below.
|
||||
- The audit trail and approval queue record which libref a change came through, so `ORDERS_EU.ORDERS` and `ORDERS_US.ORDERS` stay distinguishable in history. For most people that is a feature - you can see which report a change originated from.
|
||||
|
||||
### The concurrency caveat
|
||||
|
||||
Data Controller serialises its writes with `mp_lockanytable`, and the control table it uses, `MPE_LOCKANYTABLE`, has a primary key of `(lock_lib, lock_ds)`. That is the *registration*, not the physical file. At approve time, `postdata` takes the lock on the base table named in the submit record, and the same service runs its "has this table been updated since the diff screen was shown" check against `MPE_DATALOADS`, keyed on that same `libref` and `dsn`.
|
||||
|
||||
Two librefs over one location are two different `(libref, dsn)` pairs, so:
|
||||
|
||||
- two approvals arriving through different reports take two different locks, and neither one blocks the other;
|
||||
- a load through one report writes its `MPE_DATALOADS` entry against its own name, so the other report's staleness check does not see it either.
|
||||
|
||||
So two people editing the same rows through different reports can both be approved, and the second write wins, silently. The lock is advisory to begin with - `mp_lockanytable` is, in its own words, "only useful if every update uses the macro" - so this is not a new class of risk, but registering one file twice is a new way to fall into it.
|
||||
|
||||
The hook route does not have this problem. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity.
|
||||
|
||||
If you do use two librefs, and concurrency matters to you, the cleanest fix is to keep the two registrations for the edit screen and give each one a `POST_EDIT_HOOK` that re-points the changeset at a single canonical registration - then every approval is keyed on the same table regardless of which report raised it. Register that canonical name as well. The alternative, an explicit shared lock taken in `PRE_APPROVE_HOOK` and released in `POST_APPROVE_HOOK`, works too, but a failed run leaves the sentinel locked and `MPE_LOCKANYTABLE` is then a table you have to unpick by hand.
|
||||
|
||||
## Option 2: an empty mirror and a pair of hook scripts
|
||||
|
||||
Sometimes two librefs over one location are not available: a database library where the platform will not let you define the same object twice, or a site where adding a library definition is a change nobody wants to make. Then you can reach the same result with a mirror table and two hook scripts.
|
||||
|
||||
The idea is that the thing Data Controller edits is not the real table at all, but an empty table of the same shape, with hooks moving data in and out of it:
|
||||
|
||||
- a `PRE_EDIT_HOOK` fills the editor with the live rows of the real table, so the mirror never has to hold a copy
|
||||
- a `POST_EDIT_HOOK` re-points the submitted changeset at the real table, so the approval is raised against the real table and the load writes there
|
||||
|
||||
The mirror exists purely to carry the rule set, and never stores any data.
|
||||
|
||||
### The pre-edit hook
|
||||
|
||||
`PRE_EDIT_HOOK` runs inside the `getdata` service, after the user's filter has been applied and the rows sorted, with the data in `work.OUT`. It may replace that dataset, which is all this needs:
|
||||
|
||||
```sas
|
||||
data work.out;
|
||||
set ORDERS.ORDERS;
|
||||
run;
|
||||
```
|
||||
|
||||
The registered table is `ORDERS.MIRROR`, which is empty - so without the hook the editor would show nothing at all. With it, the grid shows the live rows:
|
||||
|
||||
|
||||
|
||||
Note the title bar: the mirror really is empty. Everything on screen came from the hook, and the grid is validated against the mirror's own rules rather than the real table's.
|
||||
|
||||
### The post-edit hook
|
||||
|
||||
This is the part that surprises people. `POST_EDIT_HOOK` runs inside the `mpe_loader` macro at submit time, on the staged rows, before the submit record is written. It cannot choose the target table directly - but at that point `LIBREF` and `DS` are still ordinary macro variables, and the submit record is built from them. Reassigning them re-points the whole changeset:
|
||||
|
||||
```sas
|
||||
data _null_;
|
||||
call symputx('libref','ORDERS');
|
||||
call symputx('ds','ORDERS');
|
||||
run;
|
||||
```
|
||||
|
||||
From that moment the changeset is an approval against `ORDERS.ORDERS`. The approver sees a diff against the real table, the load writes to the real table, and the mirror is never touched.
|
||||
|
||||
|
||||
|
||||
### The detail that will bite you
|
||||
|
||||
Use `call symputx`, not `%let`. `LIBREF` and `DS` are not declared `%local` in `mpe_loader`, and the hook is included into that scope - so `call symputx` finds the existing variable and updates it, while a `%let` creates a new variable in the hook's own scope and is silently discarded. The hook runs, the log looks clean, and the changeset goes to the mirror anyway.
|
||||
|
||||
### Other things to watch
|
||||
|
||||
- Concurrency behaves correctly here. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity - unlike the two-libref route above.
|
||||
- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big". [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter.
|
||||
- The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load.
|
||||
- The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table.
|
||||
- The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical.
|
||||
- At approval time the access checks run against the real table, so editors need `EDIT` on the mirror while approvers need `EDIT` and `APPROVE` on the real table.
|
||||
|
||||
## Which should you use?
|
||||
|
||||
If you can define two librefs over the same data, do that. It is configuration only, it leaves every other behaviour of the editor untouched, and there is nothing to maintain.
|
||||
|
||||
Reach for the hook scripts when the platform will not let you duplicate the library definition, or when you specifically want the rule set to be a property of the application rather than of the data.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
@@ -14,6 +14,9 @@ nav:
|
||||
- Table Viewer: dcu-tableviewer.md
|
||||
- ViewBoxes: viewboxes.md
|
||||
- Admin Services: admin-services.md
|
||||
- Developer Guide:
|
||||
- Filter in a Pre-Edit Hook: developer-guide/filter-pre-edit-hook.md
|
||||
- One Table, Two Rule Sets: developer-guide/two-rule-sets-one-table.md
|
||||
- Table Guide:
|
||||
- MPE_AUDIT: tables/mpe_audit.md
|
||||
- MPE_COLUMN_LEVEL_SECURITY: tables/mpe_column_level_security.md
|
||||
|
||||
Reference in new issue
Block a user