diff --git a/content/blog/bitemporal-historisation-and-the-sas-dds/index.md b/content/blog/bitemporal-historisation-and-the-sas-dds/index.md
index 766b332..72c1abd 100644
--- a/content/blog/bitemporal-historisation-and-the-sas-dds/index.md
+++ b/content/blog/bitemporal-historisation-and-the-sas-dds/index.md
@@ -38,9 +38,9 @@ An emerging consensus from the datawarehousing domain is the use of bitemporal d
## Background to Bitemporal
-The concept of Bi-Temporal Historisation is not new – it was originally associated with a chap called Richard Snodgrass back in 1992. There is further info on Wikipedia and this blog, and a more recent article on medium.
+The concept of Bi-Temporal Historisation is not new – it was originally associated with a chap called Richard Snodgrass back in 1992. There is further info on Wikipedia and this blog, and a more recent article on medium.
-Teradata have specifically implemented temporal features, which (interestingly) holds the datetime pairs in a single column (see attachment). Notice the SAS DDS and Teradata nomenclature differences (for SAS DDS: VALID typically means Transaction Datetimes; for Teradata: VALID refers to Business Datetimes).
Furthermore, this SUGI paper ( http://www2.sas.com/proceedings/sugi29/110-29.pdf) covers the issue. Here is an extract (page 8):
Versioning history (Type Two style) will always require at least a single updated date for the record, and two valid from / valid to dates if using a normalized data model. You will also require two dates in a star schema if past point in-time history queries are to be easily run in a single query.
Business history may also dictate a need for effective from / effective to dates when these may differ from the data warehouse versioning dates. This is especially true in certain sectors, like insurance, where value of business is counted over a period rather than a single time. It is also common when such changes are ‘forward-dated’ in operational systems.
So – enough of the background – what on earth is “Bitemporal Historisation” anyway?
Bitemporal Historisation - Overview
Once you ‘get it’, the approach is conceptually very simple. There are essentially just TWO datetime pairs to consider:
1 – Transaction datetimes. These from/to datetimes show when the warehouse table is populated. They effectively constitute a ‘version number’ for the data. If we want the latest ‘version’ of data, we query using a ‘high datetime’. If we want the version of data which existed yesterday, we query using yesterday’s date. This datetime-pair provides full auditability of results. Note that 99.999% of the time we will always query using the high datetime (current version, latest transactions). This is a standard SCD type 2 loading process. EVERY table must have these datetimes. The rest of this document will use the term 'Transaction' datetimes, to denote when the record was actually transacted, or committed, to the database.
2 – Business datetimes. These from/to datetimes show the period to which the data actually relates. NOT every table will have these datetimes – for many queries we are happy to use the current version of, say, a mapping table, even when producing results for historical periods. The rest of this document will use the term ‘Business’ dates.
Bitemporal Historisation in Detail
The concept of bitemporal historisation relates to the approach of storing both both business (real world) history alongside transaction (version) history in the same table. This approach aims to achieve the following goals: - Queries always produce the same result, even if the data changes
- Data changes can be audited. Each change is traceable.
- Existing queries can be easily adapted by adding temporal ranges to the WHERE clause.
- Programming overhead is reasonable.
- Historisation rules can be understood by business users.
The goal of query repeatability is particularly important in regulated environments. In order to repeat results reliably, two time points must always be included in the query. An example might be:
“Which motorcyle coverage did Miss Careful have on April 1st, 2020 as we knew it on April 3rd, 2020?”
- The first date pair represents a business coverage period. This implies that each coverage must have a BusinessFrom and BusinessTo datetime to show when the coverage starts and finishes.
- The second date represents the point in time at which we made (or would have made) the query. Being a snapshot of the database contents, this is termed the “transaction” date. The rest of this document will use TransactionFrom and TransactionTo for column names.
The above query translated into Bitemporal format might look like this:
+Teradata have specifically implemented temporal features, which (interestingly) holds the datetime pairs in a single column (see attachment). Notice the SAS DDS and Teradata nomenclature differences (for SAS DDS: VALID typically means Transaction Datetimes; for Teradata: VALID refers to Business Datetimes).
Furthermore, this SUGI paper ( http://www2.sas.com/proceedings/sugi29/110-29.pdf) covers the issue. Here is an extract (page 8):
Versioning history (Type Two style) will always require at least a single updated date for the record, and two valid from / valid to dates if using a normalized data model. You will also require two dates in a star schema if past point in-time history queries are to be easily run in a single query.
Business history may also dictate a need for effective from / effective to dates when these may differ from the data warehouse versioning dates. This is especially true in certain sectors, like insurance, where value of business is counted over a period rather than a single time. It is also common when such changes are ‘forward-dated’ in operational systems.
So – enough of the background – what on earth is “Bitemporal Historisation” anyway?
Bitemporal Historisation - Overview
Once you ‘get it’, the approach is conceptually very simple. There are essentially just TWO datetime pairs to consider:
1 – Transaction datetimes. These from/to datetimes show when the warehouse table is populated. They effectively constitute a ‘version number’ for the data. If we want the latest ‘version’ of data, we query using a ‘high datetime’. If we want the version of data which existed yesterday, we query using yesterday’s date. This datetime-pair provides full auditability of results. Note that 99.999% of the time we will always query using the high datetime (current version, latest transactions). This is a standard SCD type 2 loading process. EVERY table must have these datetimes. The rest of this document will use the term 'Transaction' datetimes, to denote when the record was actually transacted, or committed, to the database.
2 – Business datetimes. These from/to datetimes show the period to which the data actually relates. NOT every table will have these datetimes – for many queries we are happy to use the current version of, say, a mapping table, even when producing results for historical periods. The rest of this document will use the term ‘Business’ dates.
Bitemporal Historisation in Detail
The concept of bitemporal historisation relates to the approach of storing both both business (real world) history alongside transaction (version) history in the same table. This approach aims to achieve the following goals: - Queries always produce the same result, even if the data changes
- Data changes can be audited. Each change is traceable.
- Existing queries can be easily adapted by adding temporal ranges to the WHERE clause.
- Programming overhead is reasonable.
- Historisation rules can be understood by business users.
The goal of query repeatability is particularly important in regulated environments. In order to repeat results reliably, two time points must always be included in the query. An example might be:
“Which motorcyle coverage did Miss Careful have on April 1st, 2020 as we knew it on April 3rd, 2020?”
- The first date pair represents a business coverage period. This implies that each coverage must have a BusinessFrom and BusinessTo datetime to show when the coverage starts and finishes.
- The second date represents the point in time at which we made (or would have made) the query. Being a snapshot of the database contents, this is termed the “transaction” date. The rest of this document will use TransactionFrom and TransactionTo for column names.
The above query translated into Bitemporal format might look like this:
SELECT coverage
@@ -48,7 +48,7 @@ FROM customer_coverage_table AS c
WHERE c.Contact_LName = 'Fudd'
AND (c.BusinessFrom le '2020-04-01:00:00:00'dt lt c.BusinessTo)
AND (c.TransactionFrom le '2020-04-03:00:00:00'dt lt c.TransactionTo);
-
Why aren't we using BETWEEN? Because between is evil!
Bitemporal Prerequisites and Implications
Implementing a bitemporal approach requires a few principles to be adopted.
Records are Never Modified
With the exception of the TransactionTo datetime field (and maybe the PROCESSED_DTTM in the DDS model), once loaded, a record must never be modified (or deleted). This would violate the objective of query repeatability.
Matching Close / Open Dates
When a transaction is closed out and re-opened, or if business values are changing over time, the closing datetime must equal the opening datetime. This is to prevent the "temporal gap" that can happen when you close out a record at, say, 23:59:59 and re-open it at 00:00:00. What happens if you query at "23:59:59.5" ? The data has disappeared!! Note - not all ETL tools have this capability. It's common for an SCD2 load to add a second, or a day, when opening new records. Business / Transaction FROM must be less than the TO value
Leading on from the previous point, FROM and TO dates cannot be equal, and it also follows that queries should be formed as follows:
+ Why aren't we using BETWEEN? Because between is evil!
Bitemporal Prerequisites and Implications
Implementing a bitemporal approach requires a few principles to be adopted.
Records are Never Modified
With the exception of the TransactionTo datetime field (and maybe the PROCESSED_DTTM in the DDS model), once loaded, a record must never be modified (or deleted). This would violate the objective of query repeatability.
Matching Close / Open Dates
When a transaction is closed out and re-opened, or if business values are changing over time, the closing datetime must equal the opening datetime. This is to prevent the "temporal gap" that can happen when you close out a record at, say, 23:59:59 and re-open it at 00:00:00. What happens if you query at "23:59:59.5" ? The data has disappeared!! Note - not all ETL tools have this capability. It's common for an SCD2 load to add a second, or a day, when opening new records. Business / Transaction FROM must be less than the TO value
Leading on from the previous point, FROM and TO dates cannot be equal, and it also follows that queries should be formed as follows:
SELECT *
FROM sometable as t
diff --git a/content/blog/data-controller-developer-perspective/index.md b/content/blog/data-controller-developer-perspective/index.md
index 1160802..4ccaa05 100644
--- a/content/blog/data-controller-developer-perspective/index.md
+++ b/content/blog/data-controller-developer-perspective/index.md
@@ -15,7 +15,7 @@ tags:
What problem does Data Controller for SAS® solve?
-
Rafal Gagor - Veteran SAS Developer
Rafal Gagor - Veteran SAS Developer
Quality of External Data
The DQ requirements for reporting are described by the 6 dimensions (Accuracy, Reliability, Completeness, Consistency, Plausibility, Timeliness), as well as the Data Quality Framework described by Patrick Hogan here and here. There are a number of 'hard checks' implemented in OneGate as part of the XBRL submissions, which are kept up to date here. However, OneGate cannot be used as a validation tool - the regulators will be monitoring the reliability of submissions by comparing the magnitude of change between resubmissions! Not to mention the data plausibility (changes in submitted values over time). Data Quality Culture
When it comes to internal processes, CRO's across Belgium must now demonstrate to accredited statutory auditors that they satisfy the 3 Principles of the circular (Governance, Technical Capacities, Process). A long list of action points are detailed - it's clear that a lot of documentation will be required to fulfil these obligations! And not only that - the documentation will need to be continually updated and maintained. It's fair to say that automated solutions have the potential to provide significant time & cost savings in this regard. Data Controller for SAS®
The Data Controller is a web based solution for capturing data from users. Data Quality is applied at source, changes are routed through an approval process before being applied, and all updates are captured for subsequent audit. The tool provides evidence of compliance with NBB_2017_27 in the following ways: Separation of Roles for Data Preparation and Validation (principle 1.2)
Data Controller differentiates between Editors (who provide the data) and Approvers (who sign it off). Editors stage data via the web interface, or by direct file upload. Approvers are then shown the new, changed, or deleted records - and can accept or reject the update. 
+When applying financial regulations in the EU (such as Solvency II, Basel III or GDPR) it is common for Member States to maintain or introduce national provisions to further specify how such rules might be applied. The National Bank of Belgium (NBB) is no stranger to this, and releases a steady stream of circulars via their website. The circular of 12th October 2017 (NBB_2017_27, Jan Smets) is particularly interesting as it lays out a number of concrete recommendations for Belgian financial institutions with regard to Data Quality - and stated that these should be applied to internal reporting processes as well as the prudential data submitted. This fact is well known by affected industry participants, who have already performed a self assessment for YE2017 and reviewed documentation expectations as part of the HY2018 submission. Quality of External Data
The DQ requirements for reporting are described by the 6 dimensions (Accuracy, Reliability, Completeness, Consistency, Plausibility, Timeliness), as well as the Data Quality Framework described by Patrick Hogan here and here. There are a number of 'hard checks' implemented in OneGate as part of the XBRL submissions, which are kept up to date here. However, OneGate cannot be used as a validation tool - the regulators will be monitoring the reliability of submissions by comparing the magnitude of change between resubmissions! Not to mention the data plausibility (changes in submitted values over time). Data Quality Culture
When it comes to internal processes, CRO's across Belgium must now demonstrate to accredited statutory auditors that they satisfy the 3 Principles of the circular (Governance, Technical Capacities, Process). A long list of action points are detailed - it's clear that a lot of documentation will be required to fulfil these obligations! And not only that - the documentation will need to be continually updated and maintained. It's fair to say that automated solutions have the potential to provide significant time & cost savings in this regard. Data Controller for SAS®
The Data Controller is a web based solution for capturing data from users. Data Quality is applied at source, changes are routed through an approval process before being applied, and all updates are captured for subsequent audit. The tool provides evidence of compliance with NBB_2017_27 in the following ways: Separation of Roles for Data Preparation and Validation (principle 1.2)
Data Controller differentiates between Editors (who provide the data) and Approvers (who sign it off). Editors stage data via the web interface, or by direct file upload. Approvers are then shown the new, changed, or deleted records - and can accept or reject the update.
Capacities established should ensure compliance in times of stress (principle 2.1)
As an Enterprise tool, the Data Controller is as scalable and resilient as your existing SAS platform.
-Capture of Errors and Inconsistencies (principle 2.2)
Data Controller has a number of features to ensure timely detection of Data Quality issues at source (such as cell validation, post edit hook scripts, duplicate removals, rejection of data with missing columns, etc etc). Where errors do make it into the system, a full history is kept (logs, copies of files etc) for all uploads and approvals. Emails of such errors can be configured for follow up. Tools and Techniques for Information Management Should be Automated (principle 2.3)
The Data Controller can be configured to execute specific .sas programs after data validation. This enables the development of a secure and integrated workflow, and helps companies to avoid the additional documentation penalties associated with "miscellaneous unconnected computer applications" and manual information processing. 
Periodic Review & Improvements (principles 2.4 and 3.4)
The Data Controller is actively maintained with the specific aim to reduce the cost of compliance with regulations such as NBB_2017_27. Our roadmap includes new features such as pre-canned reports, version 'signoff', and the ability to reinstate previous versions of data. A process for correction and final validation of reporting before submission (3.1)
As a primary and dedicated tool for data corrections, Data Controller can be described once and used everywhere. List of Divisions Involved in Preparing Tables (principle 3.2)
By using the Data Controller in combination with knowledge of data lineage (eg from SAS metadata or manual lookup table) it becomes possible to produce an automated report to identify exactly who - and hence which division - was involved in both the preparation and the validation of the all source data per reporting table for each reporting cycle. Processes should integrate and document key controls (principle 3.3)
Data Controller can be used as a staging point for verifying the quality of data, eg when data from one department must be passed to another department for processing. The user access policy will be as per the existing policy for your SAS environment. Summary
Whilst the circular provides valuable clarity on the expectations of the NBB, there are significant costs involved to prepare for, and maintain, compliance with the guidance. This is especially the case where reporting processes are disparate, and make use of disconnected EUCs and manual processes. The Data Controller for SAS® addresses and automates a number of pain points as specifically described in the circular. It is a robust and easy-to-use tool, actively maintained and documented, and provides an integrated solution on a tried and trusted platform for data management.
+Capture of Errors and Inconsistencies (principle 2.2)
Data Controller has a number of features to ensure timely detection of Data Quality issues at source (such as cell validation, post edit hook scripts, duplicate removals, rejection of data with missing columns, etc etc). Where errors do make it into the system, a full history is kept (logs, copies of files etc) for all uploads and approvals. Emails of such errors can be configured for follow up. Tools and Techniques for Information Management Should be Automated (principle 2.3)
The Data Controller can be configured to execute specific .sas programs after data validation. This enables the development of a secure and integrated workflow, and helps companies to avoid the additional documentation penalties associated with "miscellaneous unconnected computer applications" and manual information processing. Periodic Review & Improvements (principles 2.4 and 3.4)
The Data Controller is actively maintained with the specific aim to reduce the cost of compliance with regulations such as NBB_2017_27. Our roadmap includes new features such as pre-canned reports, version 'signoff', and the ability to reinstate previous versions of data. A process for correction and final validation of reporting before submission (3.1)
As a primary and dedicated tool for data corrections, Data Controller can be described once and used everywhere. List of Divisions Involved in Preparing Tables (principle 3.2)
By using the Data Controller in combination with knowledge of data lineage (eg from SAS metadata or manual lookup table) it becomes possible to produce an automated report to identify exactly who - and hence which division - was involved in both the preparation and the validation of the all source data per reporting table for each reporting cycle. Processes should integrate and document key controls (principle 3.3)
Data Controller can be used as a staging point for verifying the quality of data, eg when data from one department must be passed to another department for processing. The user access policy will be as per the existing policy for your SAS environment. Summary
Whilst the circular provides valuable clarity on the expectations of the NBB, there are significant costs involved to prepare for, and maintain, compliance with the guidance. This is especially the case where reporting processes are disparate, and make use of disconnected EUCs and manual processes. The Data Controller for SAS® addresses and automates a number of pain points as specifically described in the circular. It is a robust and easy-to-use tool, actively maintained and documented, and provides an integrated solution on a tried and trusted platform for data management.
diff --git a/content/blog/der-touristik/index.md b/content/blog/der-touristik/index.md
index f68bfde..cca21eb 100644
--- a/content/blog/der-touristik/index.md
+++ b/content/blog/der-touristik/index.md
@@ -13,4 +13,4 @@ tags:
- Use Cases
---
-We caught up with Herbert Grossmann of DER Touristik to understand how Data Controller for SAS is used within the BICC and the types of challenges it solves. 
The previous article in this series can be found here. Guten Tag, Herby! Can you tell us about your role within DER Touristik?
Yes, I am working here as project manager for BI and Analytics and my department is the BICC (Business Intelligence Competence Centre), and we have an absolute focus on the SAS technology stack - so that’s my daily business. Great. And, I understand you guys are using Data Controller for SAS. What do you use it for?
Well, mainly for managing control tables, that we have a lot of nowadays, in the data warehouse. But we also implemented what we call an "early bird booking system". There we have used the Approval process within Data Controller, which is excellent, because users, business departments etc, can approve data that would normally only be accessible within the back-end warehouse itself. So now they have an interface, which limits their access to specific views, and this is very useful - it was also highly commended by our management. So, business users can approve modifications to secure warehouse tables without having direct write-access themselves?
Exactly Fantastic. Next question. How does having Data Controller make your life easier?
Well - there is the version control of course, that gives us a much better traceability of changes to see what was changed by whom, at what time. And we have the immediate constraint checking which is also very useful because some of the tables are sensitive towards, let’s say, the changes of the primary key. And in the past when we did it the "old fashioned way" it was possible that by mistake that someone could cause duplicate primary keys or stuff like that, so this is now not possible anymore, which is very good. And like the example that I mentioned before, that now we can grant access to certain sensitive tables even for business users that would normally have no access, but we can decide whether to give them at least the right to view these tables, or during special events edit tables, or approve edits of those tables. So this gives a lot of opportunities, and makes it much easier than it was in the past. Nice! And so, talking about the past, before you had Data Controller, how did you manage modifications to data in SAS?
We classically used two approaches - on one hand using SAS Enterprise Guide to directly edit tables or do imports, such as imports of excel sheets for example. On the other hand, we have some batch processes that also do imports of Excel tables or CSV tables. So those were the classic and standard ways. And of course especially the batch one we are still using for some files, depending on the situation. But we do no editing of tables directly with Enterprise Guide anymore because it is much safer and easier to use the Data Controller. Understood. So on the Data Controller side, what would you say were your favourite features and why?
I would say that I like the editor as a whole very much. I think that is great that in the moment you make a table editable, you can define the ways in which you would edit the tables. Like whether there is some historic logging or not, and the fact you can set the constraints. And in the editor then you have a lot of Data Quality opportunities such as defining drop-down lists for certain attributes, which really makes editing the tables easier and much more comfortable. It was a little bit of a pain in the past but now it’s almost fun. That's great feedback! Is there anything else, any comments you would like to add?
Yes, I like the fact that Data Controller is really just a part of the SAS environment. It’s not a completely separate application that you have to install somewhere, but a kind of pluggable part of the SAS environment. I liked it very much because then you still have everything in your hands. I mean I am not a developer but my knowledge of SAS is already enough to match the criteria to be able to handle the Data Controller as whole, to even do the updates and/or to modify things. And also it’s easy to show others who have experience with SAS how the tool works and what is to be done when there are data issues. And yeah, I think that’s a big advantage. 
+We caught up with Herbert Grossmann of DER Touristik to understand how Data Controller for SAS is used within the BICC and the types of challenges it solves. The previous article in this series can be found here. Guten Tag, Herby! Can you tell us about your role within DER Touristik?
Yes, I am working here as project manager for BI and Analytics and my department is the BICC (Business Intelligence Competence Centre), and we have an absolute focus on the SAS technology stack - so that’s my daily business. Great. And, I understand you guys are using Data Controller for SAS. What do you use it for?
Well, mainly for managing control tables, that we have a lot of nowadays, in the data warehouse. But we also implemented what we call an "early bird booking system". There we have used the Approval process within Data Controller, which is excellent, because users, business departments etc, can approve data that would normally only be accessible within the back-end warehouse itself. So now they have an interface, which limits their access to specific views, and this is very useful - it was also highly commended by our management. So, business users can approve modifications to secure warehouse tables without having direct write-access themselves?
Exactly Fantastic. Next question. How does having Data Controller make your life easier?
Well - there is the version control of course, that gives us a much better traceability of changes to see what was changed by whom, at what time. And we have the immediate constraint checking which is also very useful because some of the tables are sensitive towards, let’s say, the changes of the primary key. And in the past when we did it the "old fashioned way" it was possible that by mistake that someone could cause duplicate primary keys or stuff like that, so this is now not possible anymore, which is very good. And like the example that I mentioned before, that now we can grant access to certain sensitive tables even for business users that would normally have no access, but we can decide whether to give them at least the right to view these tables, or during special events edit tables, or approve edits of those tables. So this gives a lot of opportunities, and makes it much easier than it was in the past. Nice! And so, talking about the past, before you had Data Controller, how did you manage modifications to data in SAS?
We classically used two approaches - on one hand using SAS Enterprise Guide to directly edit tables or do imports, such as imports of excel sheets for example. On the other hand, we have some batch processes that also do imports of Excel tables or CSV tables. So those were the classic and standard ways. And of course especially the batch one we are still using for some files, depending on the situation. But we do no editing of tables directly with Enterprise Guide anymore because it is much safer and easier to use the Data Controller. Understood. So on the Data Controller side, what would you say were your favourite features and why?
I would say that I like the editor as a whole very much. I think that is great that in the moment you make a table editable, you can define the ways in which you would edit the tables. Like whether there is some historic logging or not, and the fact you can set the constraints. And in the editor then you have a lot of Data Quality opportunities such as defining drop-down lists for certain attributes, which really makes editing the tables easier and much more comfortable. It was a little bit of a pain in the past but now it’s almost fun. That's great feedback! Is there anything else, any comments you would like to add?
Yes, I like the fact that Data Controller is really just a part of the SAS environment. It’s not a completely separate application that you have to install somewhere, but a kind of pluggable part of the SAS environment. I liked it very much because then you still have everything in your hands. I mean I am not a developer but my knowledge of SAS is already enough to match the criteria to be able to handle the Data Controller as whole, to even do the updates and/or to modify things. And also it’s easy to show others who have experience with SAS how the tool works and what is to be done when there are data issues. And yeah, I think that’s a big advantage.
diff --git a/content/blog/euc-management-system/index.md b/content/blog/euc-management-system/index.md
index 6801e5b..91813c0 100644
--- a/content/blog/euc-management-system/index.md
+++ b/content/blog/euc-management-system/index.md
@@ -16,14 +16,16 @@ tags:
---
End User Computing (EUC) applications are unavoidable - the challenge is not to erase them, but to embrace automated approaches to EUC management that will identify, clean, secure, backup, and integrate EUC data with full auditability, ownership, and approval.
+
The Much-Maligned EUC
EUC applications such as Excel, Access Databases, and locally executed programs, are often targeted as the source of a myriad of risks - such as financial misstatements, internal fraud, incorrect models, and potential for business process disruption. The rationale being that business developed / owned applications are not subject to the same access controls, development & testing standards, documentation and release management processes as can be found over the "IT Fence". Whilst this is probably true, the inherent flexibility of EUCs that can be quickly updated without service desk requests, project codes, or lost arms & legs - means that EUCs are, regardless, here to stay.
The challenge is to find a way to shine a light onto this "Shadow IT", and provide a framework by which EUC data can be extracted in a simple, safe, secure, scalable, and auditable fashion. 
+
EUCs can be Controlled
-The 'war on EUCs' cannot be won - it simply isn't practical to ban them, or to migrate / redevelop every closely held and highly complex legacy VBA application. Until alternative solutions for Citizen Developers to build Enterprise Apps (such as SASjs) become mainstream, simple measures / controls on the EUCs themselves must be implemented - such as version control, readonly attributes, embedded documentation, peer review etc. In the meantime, a management system for EUCs is the ideal place for capturing the requisite metadata needed to monitor, audit, and secure the data therein. Such a management system should have, as a minimum, the following attributes:
+The 'war on EUCs' cannot be won - it simply isn't practical to ban them, or to migrate / redevelop every closely held and highly complex legacy VBA application. Until alternative solutions for Citizen Developers to build Enterprise Apps (such as SASjs) become mainstream, simple measures / controls on the EUCs themselves must be implemented - such as version control, readonly attributes, embedded documentation, peer review etc. In the meantime, a management system for EUCs is the ideal place for capturing the requisite metadata needed to monitor, audit, and secure the data therein. Such a management system should have, as a minimum, the following attributes:
EUC Data Quality at Source
-The ability to run data quality routines at the point of data upload (from EUC to secure IT environment) provides instant feedback to EUC operators that will allow them to make corrections and avoid costly post-upload investigations, re-runs, or worse - incorrect results. As part of this process, it should be easy to create and update those Data Quality rules. A longer discussion of Data Quality can be found here.
+The ability to run data quality routines at the point of data upload (from EUC to secure IT environment) provides instant feedback to EUC operators that will allow them to make corrections and avoid costly post-upload investigations, re-runs, or worse - incorrect results. As part of this process, it should be easy to create and update those Data Quality rules. A longer discussion of Data Quality can be found here.
EUC Data Review (4 eyes)
After EUC data is submitted, it should be reviewed before the target database is updated. It should be possible (but not mandatory) for this check to be performed by a different individual. When performing that check, it should only be necessary to review new / changed / deleted records. For changed records, the reviewer should also be able to see the original values. If the data is approved, the target table is updated. If rejected, the staged data can simply be archived.
Roles & Responsibilities (RACI)
diff --git a/content/blog/sarbanes-oxley/index.md b/content/blog/sarbanes-oxley/index.md
index d803dd5..66b3445 100644
--- a/content/blog/sarbanes-oxley/index.md
+++ b/content/blog/sarbanes-oxley/index.md
@@ -23,9 +23,9 @@ There are many aspects to full Sarbanes-Oxley (SOX) compliance, the [legislation
Data Controller facilitates internal controls through a 4 eyes review & approve mechanism for data changes. This, combined with data validation and an integrated workflow feature, provides a mechanism to easily track and report on the number of internal controls (quality rules, signoffs, rejections), as well as the frequency they are applied, who is applying them, which data items the controls relate to, and who is performing them. Such metrics can be compared and contrasted with pre-existing and current quality measures to help determine control effectiveness. Variations in the number of submit / approve cycles between reporting teams, also provide objective and repeatable measurements to support the assessment of the effectiveness of internal controls.
-
Sarbanes OxleySarbanes Oxley
- Section 404 is widely considered the most onerous part of Sarbanes-Oxley, as the documentation and testing of all the controls requires significant time and effort. To address this, the Public Company Accounting Oversight Board (PCAOB - a US non-profit created by the Sarbanes-Oxley act itself) released additional guidance to assist management and auditors in producing their reports. This is officially labeled "Auditing Standard No. 5 - An Audit of Internal Control Over Financial Reporting That Is Integrated with An Audit of Financial Statements" A few points are highlighted by the guidance in this standard that are pertinent to users of Data Controller. PCAOB AS5 Sec24 - Controls Over Management Override
Management Overrides (the freedom to simply "replace" reporting figures based on, presumably, sound judgement) are entity level controls that can be easily captured (in a centralised manner) by Data Controller. This in fact, is the "core functionality" of the tool. Data Stewards / Data Processors (Editors) make the change, then one or more Data Owners / Data Controllers (Approvers) sign it off before it is applied to the target table. A copy of the original excel file (if used) and a record of who made the change, when, what the change was, and why (if a reason is provided) is recorded. Data Validation rules can also be defined to ensure that inputs fit the desired pattern(s). 
For fun, we made a short video for this part:
+ Section 404 is widely considered the most onerous part of Sarbanes-Oxley, as the documentation and testing of all the controls requires significant time and effort. To address this, the Public Company Accounting Oversight Board (PCAOB - a US non-profit created by the Sarbanes-Oxley act itself) released additional guidance to assist management and auditors in producing their reports. This is officially labeled "Auditing Standard No. 5 - An Audit of Internal Control Over Financial Reporting That Is Integrated with An Audit of Financial Statements" A few points are highlighted by the guidance in this standard that are pertinent to users of Data Controller. PCAOB AS5 Sec24 - Controls Over Management Override
Management Overrides (the freedom to simply "replace" reporting figures based on, presumably, sound judgement) are entity level controls that can be easily captured (in a centralised manner) by Data Controller. This in fact, is the "core functionality" of the tool. Data Stewards / Data Processors (Editors) make the change, then one or more Data Owners / Data Controllers (Approvers) sign it off before it is applied to the target table. A copy of the original excel file (if used) and a record of who made the change, when, what the change was, and why (if a reason is provided) is recorded. Data Validation rules can also be defined to ensure that inputs fit the desired pattern(s). For fun, we made a short video for this part:
`youtube: https://youtu.be/iY3KQZL4ok0`
@@ -39,7 +39,7 @@ Below is an example of column level lineage. Like Table Level lineage, this can
The ability to define additional data lineages, outside of SAS (eg between spreadsheets or other reporting systems) is in the product roadmap, along with lineage from SAS Viya. PCAOB AS5 App B - Benchmarking of Automated Controls
The use of IT secured financial controls can significantly reduce the cost of Sarbanes-Oxley compliance testing following the first year assessment, particularly where the source code is secured and cannot be modified by users. The core programs (services) within the Data Controller application that perform data signoffs are mature, distinct and change tracked - so it is possible for Data Controller to be upgraded in-place without affecting the benchmarking strategy. This contrasts with spreadsheet based control mechanisms, which must be revalidated in each reporting period.
-
PCAOB Release 2007-005A, Appendix BPCAOB Release 2007-005A, Appendix B
## Sarbanes-Oxley Act Section 1102 - Tampering
diff --git a/content/blog/siemens-healthineers-smart-data-catalog/index.md b/content/blog/siemens-healthineers-smart-data-catalog/index.md
index f8dcab6..76a9693 100644
--- a/content/blog/siemens-healthineers-smart-data-catalog/index.md
+++ b/content/blog/siemens-healthineers-smart-data-catalog/index.md
@@ -16,7 +16,7 @@ tags:
- VBA
---
-Data Controller was implemented at Siemens Healthineers to facilitate their SAS-Powered Smart Data Catalog and integrate with Data Lineage reporting. We are grateful to Helen Stark (Power User) and Hans-Juergen Kopperger (SAS App Developer) for sharing their "before and after" experiences. The previous article in this series is available here.
+Data Controller was implemented at Siemens Healthineers to facilitate their SAS-Powered Smart Data Catalog and integrate with Data Lineage reporting. We are grateful to Helen Stark (Power User) and Hans-Juergen Kopperger (SAS App Developer) for sharing their "before and after" experiences. The previous article in this series is available here.
---
@@ -76,7 +76,7 @@ In the past we had a custom Stored Process web app for uploading excel files, ba
I would often receive support tickets in relation to this upload, the cause of which was often due to the diversity of our excel templates, and being unsure which was the right template...So, we would have a lot of discussions about how to bring data into the backend in a controlled manner.
-Then one day, I got information through SAS User Group Germany that you provide a solution with Data Controller. I was initially interested in the Data Lineage functionality, but then I understood the main concept behind Data Controller. And for me the main benefit is that I can save a lot of time - with out of the box features like the web data editor, and the web upload facility with excel spreadsheet drag and drop. And there is the automatic workflow behind with the mandatory approval step. Since we implemented Data Controller, we no longer get those support tickets.
+Then one day, I got information through SAS User Group Germany that you provide a solution with Data Controller. I was initially interested in the Data Lineage functionality, but then I understood the main concept behind Data Controller. And for me the main benefit is that I can save a lot of time - with out of the box features like the web data editor, and the web upload facility with excel spreadsheet drag and drop. And there is the automatic workflow behind with the mandatory approval step. Since we implemented Data Controller, we no longer get those support tickets.
### Fantastic. If you had to pick your top features, what would they be?
@@ -84,7 +84,7 @@ The main benefit is getting data controlled, and into the backend. The controlle
The transparency of the history page is another benefit. I can look at every requested submit or approval - what changes have been applied, what changes have been submitted, and what changes have been approved. This helps us a lot to get data transparency.
-The email alerts is a great feature. For the communication of changes, we had previously created a team's collaboration chat. e.g. if someone did a change and needed to request an approval. But with email alerts, the notification of changes is now automatically sent to the responsible data owner, who can immediately click the email link and do his approval. This speeds up the whole process.
+The email alerts is a great feature. For the communication of changes, we had previously created a team's collaboration chat. e.g. if someone did a change and needed to request an approval. But with email alerts, the notification of changes is now automatically sent to the responsible data owner, who can immediately click the email link and do his approval. This speeds up the whole process.
Another advantage is the "database approach" for updates. So, someone is changing one row in a table which is connected to his use case, another guy can change other rows of the same table, nearly simultaneously. Because not everyone is changing the same rows. Everyone has their own subset of rows, their own "workspace" within one table. In the past we would have one excel template, and this would always override all values. We would have a lot of excel templates going around our colleagues, so there were always conflicts of overrides and versioning, and stuff like that. With Data Controller, it's now a simple, easy and transparent data capture process.
diff --git a/content/blog/v6-0-api-explorer/index.md b/content/blog/v6-0-api-explorer/index.md
index 9f03d1a..7624d0d 100644
--- a/content/blog/v6-0-api-explorer/index.md
+++ b/content/blog/v6-0-api-explorer/index.md
@@ -1,5 +1,5 @@
---
-title: "v6.0 Release: Viya API Explorer"
+title: 'v6.0 Release: Viya API Explorer'
description: Data Controller community tier now includes an API explorer! We've also overhauled the (in)format ingestion capability, and revamped our pricing (now with unlimited users across all tiers).
date: '2023-06-26 09:00:00'
author: 'Allan Bowe'
@@ -15,13 +15,12 @@ With a (Viya) API explorer, an overhauled (in)format ingestion capability, and n
## Viya API Explorer
-Following on from the metadata explorer (SAS 9 EBI feature) we have been looking to provide a similar capability for Viya. And so, we built the API explorer!
+Following on from the metadata explorer (SAS 9 EBI feature) we have been looking to provide a similar capability for Viya. And so, we built the API explorer!
-This lets you easily trigger the (GET) APIs and explore the responses without having to break open Postman or another development toolkit. Here's an example of opening a Job and examining the SAS code:
+This lets you easily trigger the (GET) APIs and explore the responses without having to break open Postman or another development toolkit. Here's an example of opening a Job and examining the SAS code:
-
Here we grab the raw JSON for pasting into VS Code:
@@ -32,29 +31,27 @@ And here we toggle the start / limit parameters to bring back more values:
We would love YOUR feedback as to how we can extend this API explorer to make it an even more useful tool!
-
-
## Unlimited Users
-If you've been following us for a while you've probably heard the '5 users free' tagline. Well - you will hear it no more, as we now offer **unlimited users for all tier levels**!
+If you've been following us for a while you've probably heard the '5 users free' tagline. Well - you will hear it no more, as we now offer **unlimited users for all tier levels**!
That's right, you can download Data Controller (Community Edition) and use it across your entire enterprise TODAY, without spending a penny.
-If, however, you would like priority support and full access to all features, we ask that you engage us on paid subscription plan.
+If, however, you would like priority support and full access to all features, we ask that you engage us on paid subscription plan.
## (IN)FORMAT Capabilities
-Previously we only supported ingestion of run-of-the-mill SAS formats. Following customer feedback, we have now expanded this capability to include:
+Previously we only supported ingestion of run-of-the-mill SAS formats. Following customer feedback, we have now expanded this capability to include:
-* Informats
-* Multilabel Formats
-* NotSorted Formats
+- Informats
+- Multilabel Formats
+- NotSorted Formats
-The addition of these format types broke the data model we were using previously for holding format data. We had incorrectly assumed that the CNTLOUT dataset could be keyed on TYPE, FMTNAME and START.
+The addition of these format types broke the data model we were using previously for holding format data. We had incorrectly assumed that the CNTLOUT dataset could be keyed on TYPE, FMTNAME and START.
-In fact, START can be null, and the format data can have complete duplicates (multilabel). Furthermore, the _order_ of records is important (notsorted). Therefore we have applied a new key (TYPE, FMTNAME, FMTROW) where FMTROW is the index of the record of the format in question.
+In fact, START can be null, and the format data can have complete duplicates (multilabel). Furthermore, the _order_ of records is important (notsorted). Therefore we have applied a new key (TYPE, FMTNAME, FMTROW) where FMTROW is the index of the record of the format in question.
-This means if you insert a row in a format, Data Controller will see this as a CHANGE to all the rows underneath (if they are not duplicates). This difference in behaviour, as well as the the change in the model, is the "breaking change" in this release (hence major version bump). It will likely only affect you though if you are using Excel or CSV to upload (in)format data.
+This means if you insert a row in a format, Data Controller will see this as a CHANGE to all the rows underneath (if they are not duplicates). This difference in behaviour, as well as the the change in the model, is the "breaking change" in this release (hence major version bump). It will likely only affect you though if you are using Excel or CSV to upload (in)format data.
This primary key (TYPE, FMTNAME, FMTROW) is now also indicated in VIEW mode.
@@ -68,7 +65,6 @@ We've added a new screen (under the username dropdown) to show system details as
This screen is also available for regular users (those not in the Data Controller admin group), just without the additional buttons.
-
## Load More Values
We've added the ability to 'load more' history on the history page, as well as the ability to [show more history by default](https://docs.datacontroller.io/dcc-options/#history_rows)
@@ -81,31 +77,14 @@ We've added the ability to 'load more' history on the history page, as well as t
Some of the issues we've zapped:
-* Enable data-catalog refresh of a single library when invalid libraries are present
-* Prevent error when attempting an UNLOCK of an already-unlocked table
-* Show Viya avatar when web app is served from a different domain
-* Bug with delete-only uploads not appearing in the audit table
-* Show special missing values on VIEW screen
+- Enable data-catalog refresh of a single library when invalid libraries are present
+- Prevent error when attempting an UNLOCK of an already-unlocked table
+- Show Viya avatar when web app is served from a different domain
+- Bug with delete-only uploads not appearing in the audit table
+- Show special missing values on VIEW screen
## Roadmap
-Looking to the future, we are actively tidying up the codebase to publish it as 'source-available' (the source is already available to existing customers). We are also investigating the HandsOnTable "Formula" feature to see if we can implement it on the EDIT grid.
+Looking to the future, we are actively tidying up the codebase to publish it as 'source-available' (the source is already available to existing customers). We are also investigating the HandsOnTable "Formula" feature to see if we can implement it on the EDIT grid.
If you would like to see any new features in DC, or would like to kick the tyres and give it a whirl, do [get in touch](https://datacontroller.io/contact)!
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/src/markdown-pages/pricing.md b/src/markdown-pages/pricing.md
index d41737e..366326e 100644
--- a/src/markdown-pages/pricing.md
+++ b/src/markdown-pages/pricing.md
@@ -15,30 +15,29 @@ description: Data Controller for SAS® pricing starts from FREE, is fully TRA
## Packages
-Data Controller is available in four packages, with no limits on number of users (since [v6](https://datacontroller.io/v6-0-api-explorer/)). Data Controller can run on ANY flavour of SAS - be that Viya, EBI, or [SASjs Server](https://server.sasjs.io).
+Data Controller is available in four packages, with no limits on number of users (since [v6](https://datacontroller.io/v6-0-api-explorer/)). Data Controller can run on ANY flavour of SAS - be that Viya, EBI, or [SASjs Server](https://server.sasjs.io).
-| | Community | Bronze | Silver | Gold |
-|--------------------------|-----------------|-----------|-----------|------------|
-| Number of Users | Unlimited | Unlimited | Unlimited | Unlimited |
-| Environments | Unlimited | Unlimited | Unlimited | Unlimited |
-| Libraries | Unlimited | Unlimited | Unlimited | Unlimited |
-| Tables (per library) | 35 | 100 | Unlimited | Unlimited |
-| Rows (VIEW) | 15 | 100 | Unlimited | Unlimited |
-| Rows (UPLOAD) | 5 | 100 | Unlimited | Unlimited |
-| Rows (HISTORY) | 15 | 100 | Unlimited | Unlimited |
-| Lineage | 3 per day | 100 | Unlimited | Unlimited |
-| Viewboxes | 1 | 3 | 6 (max) | 6 (max) |
-| Data Controller Support | Best Endeavours | 8hr SLA | 8hr SLA | 4hr SLA |
-| SASjs Support | ❌ | ✅ | ✅ | ✅ |
-| In-house SAS App Support | ❌ | ❌ | ✅ | ✅ |
-| Developer Days | ❌ | ❌ | ❌ | ✅ |
-| Consulting Days | ❌ | ❌ | ❌ | ✅ |
-| Perpetual Licence Option | ❌ | ❌ | ❌ | ✅ |
-| Price | FREE | 12k EUR pa | 25k EUR pa| Contact Us |
+| | Community | Bronze | Silver | Gold |
+| ------------------------ | --------------- | ---------- | ---------- | ---------- |
+| Number of Users | Unlimited | Unlimited | Unlimited | Unlimited |
+| Environments | Unlimited | Unlimited | Unlimited | Unlimited |
+| Libraries | Unlimited | Unlimited | Unlimited | Unlimited |
+| Tables (per library) | 35 | 100 | Unlimited | Unlimited |
+| Rows (VIEW) | 15 | 100 | Unlimited | Unlimited |
+| Rows (UPLOAD) | 5 | 100 | Unlimited | Unlimited |
+| Rows (HISTORY) | 15 | 100 | Unlimited | Unlimited |
+| Lineage | 3 per day | 100 | Unlimited | Unlimited |
+| Viewboxes | 1 | 3 | 6 (max) | 6 (max) |
+| Data Controller Support | Best Endeavours | 8hr SLA | 8hr SLA | 4hr SLA |
+| SASjs Support | ❌ | ✅ | ✅ | ✅ |
+| In-house SAS App Support | ❌ | ❌ | ✅ | ✅ |
+| Developer Days | ❌ | ❌ | ❌ | ✅ |
+| Consulting Days | ❌ | ❌ | ❌ | ✅ |
+| Perpetual Licence Option | ❌ | ❌ | ❌ | ✅ |
+| Price | FREE | 12k EUR pa | 25k EUR pa | Contact Us |
-Data Controller Development Days (to build new product features etc) are available for all plans at a price of 600 EUR per day. Consulting rates are published here.
-
+Data Controller Development Days (to build new product features etc) are available for all plans at a price of 600 EUR per day. Consulting rates are published here.
## Support
@@ -48,10 +47,9 @@ Our support goal is to get you back up and running as soon as possible in the ca
Tickets may be raised at either:
-1. Support Desk: https://support.4gl.io
+1. Support Desk: https://support.4gl.io
2. Support Mailbox: support@datacontroller.io
-
The corresponding severity level should also be specified, eg as follows:
1. Level 1 - App is non-functional and cannot be used
@@ -83,22 +81,21 @@ The following items are assumed:
4. The client is able to provide relevant artefacts (logs, sample data, clarifications)
### Escalation
+
If a ticket cannot be resolved within the agreed SLA conditions, escalation has to be started by the assigned support department. Escalation will be made via video call between managers at both the client and 4GL.
Complaints on services should be directed to both the 4GL Engagement Manager and the client Application Owner.
-
### In Scope Support
Support is unlimited, but restricted to items that actually relate to the tool (eg we cannot help with permissions, or data issues, beyond the guidance in the documentation). All support is provided remotely, using video calls / screen sharing technology.no
-
Examples of ‘in-scope’ (not an exhaustive list):
-* configuration & usage advice
-* troubleshooting
-* installation / deployment support
-* integration advice (actual integrations are a consulting effort)
+- configuration & usage advice
+- troubleshooting
+- installation / deployment support
+- integration advice (actual integrations are a consulting effort)
### Out of Scope Support
@@ -134,4 +131,3 @@ There may be situations where support cannot give immediate fixes, such as (but
### Partner Purchases
Where Data Controller is purchased directly from one of our partners, the partner may provide first line support as per your agreement with them.
-
diff --git a/src/pages/about.tsx b/src/pages/about.tsx
index 9b1dda1..ddac12b 100644
--- a/src/pages/about.tsx
+++ b/src/pages/about.tsx
@@ -132,7 +132,12 @@ const About: React.FC> = ({ data, location }) => {
The company supports the development of the Data Controller web
application and other{' '}
-
+
SAS Apps
, and the Macro Cards range of websites listed below: