From 971b007606b0a363a1a49cd5c42d0c8d178377f6 Mon Sep 17 00:00:00 2001 From: dc Date: Thu, 22 Oct 2026 16:40:48 +0000 Subject: [PATCH 1/3] docs(options): the EDIT limits truncate the grid and warn getdata no longer refuses a selection over DC_MAXOBS_WEBEDIT or DC_MAXCELLS_WEBEDIT - it serves the rows within the limit and flags the truncation against the table name. Update the option reference and the two developer-guide pages that claimed the edit screen stops with "Table is too big". The hook page also carried the old 500 row default. --- docs/dcc-options.md | 2 ++ docs/developer-guide/filter-pre-edit-hook.md | 2 +- docs/developer-guide/two-rule-sets-one-table.md | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/dcc-options.md b/docs/dcc-options.md index 87a3805..50d06f0 100644 --- a/docs/dcc-options.md +++ b/docs/dcc-options.md @@ -23,6 +23,8 @@ Set to YES or NO to enable email alerts. This requires email options to be preco ### DC_MAXCELLS_WEBEDIT By default, a maximum of 200,000 cells (observations multiplied by variables) can be edited in the browser at one time. This limit is applied together with [DC_MAXOBS_WEBEDIT](#dc_maxobs_webedit) - whichever is reached first. +A selection over either limit is served truncated: the grid carries the rows that fit, and a warning triangle next to the table name names the limit that was reached. Filter the table to see the rest. + The cell count matters because the cost of an EDIT screen load tracks the number of cells, not the number of rows. A 10 column table is comfortable at 250 rows, but 250 rows of a 1000 column table is 250,000 cells. ### DC_MAXOBS_WEBEDIT diff --git a/docs/developer-guide/filter-pre-edit-hook.md b/docs/developer-guide/filter-pre-edit-hook.md index 2ee4a59..312ab6d 100644 --- a/docs/developer-guide/filter-pre-edit-hook.md +++ b/docs/developer-guide/filter-pre-edit-hook.md @@ -146,7 +146,7 @@ The user's clause and Data Controller's clause have to be kept apart, because Ro ## The row cap -`getdata` refuses a result larger than `DC_MAXOBS_WEBEDIT` (500 by default) with *Table is too big*. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`. +`getdata` truncates a result larger than `DC_MAXOBS_WEBEDIT` (250 by default) or `DC_MAXCELLS_WEBEDIT` to the rows within the limit, and warns against the table name. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap - the rows past the limit never reach the browser. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`. ## Matching the filter chips diff --git a/docs/developer-guide/two-rule-sets-one-table.md b/docs/developer-guide/two-rule-sets-one-table.md index e42ef8a..ddfa32a 100644 --- a/docs/developer-guide/two-rule-sets-one-table.md +++ b/docs/developer-guide/two-rule-sets-one-table.md @@ -109,7 +109,7 @@ From that moment the changeset is an approval against `ORDERS.ORDERS`. The appro ### Other things to watch - Concurrency behaves correctly here. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity - unlike the two-libref route above. -- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap will stop the edit screen with "Table is too big". [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter. +- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap truncates the edit screen, with a warning against the table name. [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter. - The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load. - The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table. - The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical. From 561e92e090b7bf06e633aedb814e45f12c59ce25 Mon Sep 17 00:00:00 2001 From: dc Date: Thu, 22 Oct 2026 19:27:51 +0000 Subject: [PATCH 2/3] docs(developer-guide): name both EDIT caps in the hook page The truncation is whichever of DC_MAXOBS_WEBEDIT and DC_MAXCELLS_WEBEDIT binds first, and for a wide table it is the cell cap - floor(cells/columns) rows, potentially far below the row limit. The hook page named only the row cap and quoted its 250 default, so a hook author would check the row count alone. Same correction in the two-rule-sets page. --- docs/developer-guide/filter-pre-edit-hook.md | 4 ++-- docs/developer-guide/two-rule-sets-one-table.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/developer-guide/filter-pre-edit-hook.md b/docs/developer-guide/filter-pre-edit-hook.md index 312ab6d..5d48f94 100644 --- a/docs/developer-guide/filter-pre-edit-hook.md +++ b/docs/developer-guide/filter-pre-edit-hook.md @@ -144,9 +144,9 @@ The captures below are the editor for a table with an `ORDER_DATE` column, opene The user's clause and Data Controller's clause have to be kept apart, because Row Level Security lives in the second one. Combining them as `( user OR previous-quarter ) AND ( validity + RLS )` keeps RLS applied to both branches. Wrapping the whole existing filter instead - `( existing OR previous-quarter )` - would let the previous-quarter rows through without their RLS condition, so do not take that shortcut. -## The row cap +## The row and cell caps -`getdata` truncates a result larger than `DC_MAXOBS_WEBEDIT` (250 by default) or `DC_MAXCELLS_WEBEDIT` to the rows within the limit, and warns against the table name. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap - the rows past the limit never reach the browser. Check the count before handing the data back, or raise the option for that table in `MPE_CONFIG`. +`getdata` truncates a result over either `DC_MAXOBS_WEBEDIT` or `DC_MAXCELLS_WEBEDIT`, whichever binds first - a wide table runs out of cells before rows, at floor(cells / columns) rows - and warns against the table name. Widening a filter widens the result, so a hook that adds a quarter can push a table over the cap: the rows past the limit never reach the browser. Check both counts before handing the data back, or raise the options for that table in `MPE_CONFIG`. ## Matching the filter chips diff --git a/docs/developer-guide/two-rule-sets-one-table.md b/docs/developer-guide/two-rule-sets-one-table.md index ddfa32a..ebc3196 100644 --- a/docs/developer-guide/two-rule-sets-one-table.md +++ b/docs/developer-guide/two-rule-sets-one-table.md @@ -109,7 +109,7 @@ From that moment the changeset is an approval against `ORDERS.ORDERS`. The appro ### Other things to watch - Concurrency behaves correctly here. Both mirrors route their submit to the same base table, so every approval locks, loads and logs against one identity - unlike the two-libref route above. -- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` cap truncates the edit screen, with a warning against the table name. [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter. +- The filter has already been applied to the empty mirror by the time the pre-edit hook runs, so a hook that reads the real table ignores the user's filter unless it re-applies it (`where %inc filtref`). On a small table you will not notice; on a large one the `DC_MAXOBS_WEBEDIT` or `DC_MAXCELLS_WEBEDIT` cap truncates the edit screen, with a warning against the table name. [Updating the Filter in a Pre-Edit Hook](/developer-guide/filter-pre-edit-hook/) works through extending that filter. - The hook's output must have the same columns the editor expects - the real table, minus any transaction or processing columns that Data Controller drops on load. - The real table must itself be registered in `MPE_TABLES`. The approval screen resolves the table's audit settings from that row, so a changeset routed to a table with no registration cannot be reviewed - the submit is refused up front, naming the table. - The mirror's `MPE_TABLES` row is read for the edit screen and the real table's for the load, so keep their `buskey`, `loadtype` and temporal column settings identical. From 856e57173fa3fd9e2bddfb0e68a9a2c2b080afdf Mon Sep 17 00:00:00 2001 From: dc Date: Mon, 26 Oct 2026 11:57:21 +0000 Subject: [PATCH 3/3] docs(overview): the EDIT limits truncate, and the viewer default is 2000 not 500 Review 431: dc-overview.md still said "Up to 500 rows can be edited (in the web editor) at one time" - the wrong number (DC_MAXOBS_WEBEDIT ships at 250) and, once dc/dc#352 lands, the wrong behaviour (an over-limit selection is served truncated with a warning, not refused or capped at a count). It now names both limits, whichever binds first, and links to the options that set them. Sweeping for the same stale figure turned up dcu-tableviewer.md, which says 500 twice and uses 500 in its worked example of the row count above the grid. DC_MAXOBS_WEBVIEW has been 2000 since 20261003_row_and_cell_limits.sas (the migration records 500 -> 2000), and mpe_makedata seeds 2000, so those three places were two releases out of date. Fixed here, and flagged in the PR reply as VIEW rather than EDIT in case it belongs in its own change. Checked: no "500 rows" or stale abort wording left in docs/, and the two anchors the overview now links to (dc_maxobs_webedit, dc_maxcells_webedit) exist on the published options page. --- docs/dc-overview.md | 4 +++- docs/dcu-tableviewer.md | 8 ++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/dc-overview.md b/docs/dc-overview.md index d2177a5..525da56 100644 --- a/docs/dc-overview.md +++ b/docs/dc-overview.md @@ -35,7 +35,9 @@ When importing an excel file, all rows are loaded into the web page. You have a A CSV upload bypasses the part where the records are loaded into the web page, and ALL rows are sent to the staging area directly. This makes it suitable for larger uploads. ## Edit Stage Approve Workflow -Up to 500 rows can be edited (in the web editor) at one time. These edits are submitted to a staging area. After one or more approvals (acceptances) the changes are applied to the source table. +The EDIT screen loads up to [DC_MAXOBS_WEBEDIT](/dcc-options/#dc_maxobs_webedit) rows (250 by default), or as many rows as fit within [DC_MAXCELLS_WEBEDIT](/dcc-options/#dc_maxcells_webedit) (200,000 cells) - whichever is reached first, so a wide table loads fewer rows than a narrow one. A larger table is not refused: it loads truncated, with a warning against the table name, and the filter is the way to reach the rest. + +These edits are submitted to a staging area. After one or more approvals (acceptances) the changes are applied to the source table. ![screenshot](img/dcu_flow.png) diff --git a/docs/dcu-tableviewer.md b/docs/dcu-tableviewer.md index 7a97ac3..95b4294 100644 --- a/docs/dcu-tableviewer.md +++ b/docs/dcu-tableviewer.md @@ -1,7 +1,7 @@ # Data Controller for SAS: Viewer The viewer screen provides a raw view of the underlying table. -Choose a library, then a table, and click view to see the first 500 rows. +Choose a library, then a table, and click view to see the first rows - up to the `DC_MAXOBS_WEBVIEW` limit, 2,000 rows by default. A filter option is provided should you wish to view a different section of rows. The following libraries will be visible: @@ -21,9 +21,9 @@ The search covers **every** column of the table at once, so you do not need to k - **Character search** (the default) is a case sensitive CONTAINS match against every character column, so `Smith` finds `Smithson` and `smith` finds `Goldsmith`, but `smith` will not find `Smithson`. - **Numeric search** (tick the Numeric box) is an exact match against every numeric column, so `42` will not find `420` or `4210`. - If a filter is applied, the search runs within the filtered rows rather than the whole table. -- The grid is capped by the [`DC_MAXOBS_WEBVIEW`](/dcc-options/#dc_maxobs_webview) option (500 rows by default). -- For a search, the cap applies to the row count as well: a search that matches more rows than the cap returns 500 of them, and the count shown next to the table name is the number of rows returned, not the total number of matches. -- Without a search, the count shown next to the table name is the full number of rows in the filtered view, so it can be higher than the number of rows displayed - a table of 1,200 rows reports `1,200 rows` above a grid holding the first 500. +- The grid is capped by the [`DC_MAXOBS_WEBVIEW`](/dcc-options/#dc_maxobs_webview) option (2,000 rows by default). +- For a search, the cap applies to the row count as well: a search that matches more rows than the cap returns 2,000 of them, and the count shown next to the table name is the number of rows returned, not the total number of matches. +- Without a search, the count shown next to the table name is the full number of rows in the filtered view, so it can be higher than the number of rows displayed - a table of 5,000 rows reports `5,000 rows` above a grid holding the first 2,000. - A search that matches nothing shows a "No data found with given conditions" panel rather than an empty grid. ![Full table search](img/full-table-search.png)