fix: open ext. links in new tab

Closes #7

content/blog/bitemporal-historisation-and-the-sas-dds/index.md

@@ -38,9 +38,9 @@ An emerging consensus from the datawarehousing domain is the use of bitemporal d
 
 ## Background to Bitemporal
 
-The concept of Bi-Temporal Historisation is not new – it was originally associated with a chap called Richard Snodgrass back in 1992. <span style="color: #000000;"><span>There is further info on </span></span><span style="color: #0000ff;"><u><a href="http://en.wikipedia.org/wiki/Temporal_database"><span>Wikipedia</span></a></u></span><span style="color: #000000;"><span> and </span></span><span style="color: #0000ff;"><u><a href="http://informix-myview.blogspot.co.uk/2012/03/bitemporal-data-is-this-next-big-thing.html"><span>this blog</span></a></u></span><span style="color: #000000;"><span>, and a more recent article on <a href="https://medium.com/kamu-data/a-brief-history-of-time-in-data-modelling-olap-systems-9032f63b8b7f">medium</a>.</span></span>
+The concept of Bi-Temporal Historisation is not new – it was originally associated with a chap called Richard Snodgrass back in 1992. <span style="color: #000000;"><span>There is further info on </span></span><span style="color: #0000ff;"><u><a target="_blank" rel="noopener" href="http://en.wikipedia.org/wiki/Temporal_database"><span>Wikipedia</span></a></u></span><span style="color: #000000;"><span> and </span></span><span style="color: #0000ff;"><u><a target="_blank" rel="noopener" href="http://informix-myview.blogspot.co.uk/2012/03/bitemporal-data-is-this-next-big-thing.html"><span>this blog</span></a></u></span><span style="color: #000000;"><span>, and a more recent article on <a target="_blank" rel="noopener" href="https://medium.com/kamu-data/a-brief-history-of-time-in-data-modelling-olap-systems-9032f63b8b7f">medium</a>.</span></span>
 
-<p class="western"><span style="color: #000000;"><span>Teradata have specifically implemented </span></span><span style="color: #0000ff;"><u><a href="smb://smteam.sas.com/DavWWWRoot/psd/rmi/Implementation%20challenges/TeradataBiTemporal.pdf"><span>temporal features</span></a></u></span><span style="color: #000000;"><span>, which (interestingly) holds the datetime </span></span><span style="color: #000000;"><span><i>pairs</i></span></span><span style="color: #000000;"><span> in a single column (see attachment). Notice the SAS DDS and Teradata nomenclature differences (for SAS DDS: VALID typically means Transaction Datetimes; for Teradata: VALID refers to Business Datetimes).</span></span></p> <a href="/wp-content/uploads/2020/08/hist-op_1.1.6_en_manual.pdf"><img class=" aligncenter" src="/wp-content/uploads/2020/08/bt.png" alt="bitemporal" width="900" height="102" /></a> <p class="western"><span style="color: #000000;"> <span>Furthermore, this SUGI paper ( </span></span><span style="color: #0000ff;"><u><a href="http://www2.sas.com/proceedings/sugi29/110-29.pdf"><span>http://www2.sas.com/proceedings/sugi29/110-29.pdf</span></a></u></span><span style="color: #000000;"><span>) covers the issue. Here is an extract (page 8): </span></span></p> <blockquote> <p class="western"><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span><b>Versioning history</b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span> (Type Two style) will always require at least a single </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>updated date </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>for the record, and two </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>valid from / valid to dates </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>if using a normalized data model. You will also require two dates in a star schema if past point in-time history queries are to be easily run in a single query.</span></span></span></span></p> <p class="western"><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span><b>Business history</b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span> may also dictate a need for </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>effective from / effective to dates </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>when these may differ from the data warehouse versioning dates. This is especially true in certain sectors, like insurance, where value of business is counted over a period rather than a single time. It is also common when such changes are ‘forward-dated’ in operational systems.</span></span></span></span></p> </blockquote> <p class="western">So – enough of the background – what on earth is “Bitemporal Historisation” anyway?</p> <h2 class="western">Bitemporal Historisation - Overview</h2> <p class="western">Once you ‘get it’, the approach is conceptually very simple. There are essentially just TWO datetime pairs to consider:</p> <p class="western"><b>1 – Transaction datetimes.</b> These from/to datetimes show when the <i>warehouse</i> table is populated. They effectively constitute a ‘version number’ for the data. If we want the latest ‘version’ of data, we query using a ‘high datetime’. If we want the version of data which existed yesterday, we query using yesterday’s date. This datetime-pair provides <i>full auditability</i> of results. Note that 99.999% of the time we will always query using the high datetime (current version, latest transactions). This is a standard SCD type 2 loading process. EVERY table must have these datetimes. The rest of this document will use the term 'Transaction' datetimes, to denote when the record was actually transacted, or committed, to the database.</p> <p class="western"><b>2 – Business datetimes</b>. These from/to datetimes show the period to which the data actually relates. NOT every table will have these datetimes – for many queries we are happy to use the <i>current</i> version of, say, a mapping table, even when producing results for historical periods.Line truncated
+<p class="western"><span style="color: #000000;"><span>Teradata have specifically implemented </span></span><span style="color: #0000ff;"><u><a target="_blank" rel="noopener" href="smb://smteam.sas.com/DavWWWRoot/psd/rmi/Implementation%20challenges/TeradataBiTemporal.pdf"><span>temporal features</span></a></u></span><span style="color: #000000;"><span>, which (interestingly) holds the datetime </span></span><span style="color: #000000;"><span><i>pairs</i></span></span><span style="color: #000000;"><span> in a single column (see attachment). Notice the SAS DDS and Teradata nomenclature differences (for SAS DDS: VALID typically means Transaction Datetimes; for Teradata: VALID refers to Business Datetimes).</span></span></p> <a href="/wp-content/uploads/2020/08/hist-op_1.1.6_en_manual.pdf"><img class=" aligncenter" src="/wp-content/uploads/2020/08/bt.png" alt="bitemporal" width="900" height="102" /></a> <p class="western"><span style="color: #000000;"> <span>Furthermore, this SUGI paper ( </span></span><span style="color: #0000ff;"><u><a target="_blank" rel="noopener" href="http://www2.sas.com/proceedings/sugi29/110-29.pdf"><span>http://www2.sas.com/proceedings/sugi29/110-29.pdf</span></a></u></span><span style="color: #000000;"><span>) covers the issue. Here is an extract (page 8): </span></span></p> <blockquote> <p class="western"><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span><b>Versioning history</b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span> (Type Two style) will always require at least a single </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>updated date </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>for the record, and two </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>valid from / valid to dates </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>if using a normalized data model. You will also require two dates in a star schema if past point in-time history queries are to be easily run in a single query.</span></span></span></span></p> <p class="western"><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span><b>Business history</b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span> may also dictate a need for </span></span></span></span><span style="color: #000000;"><span style="font-family: Arial-BoldMT, Arial Bold, sans-serif;"><span style="font-size: small;"><span><b>effective from / effective to dates </b></span></span></span></span><span style="color: #000000;"><span style="font-family: ArialMT, Arial, sans-serif;"><span style="font-size: small;"><span>when these may differ from the data warehouse versioning dates. This is especially true in certain sectors, like insurance, where value of business is counted over a period rather than a single time. It is also common when such changes are ‘forward-dated’ in operational systems.</span></span></span></span></p> </blockquote> <p class="western">So – enough of the background – what on earth is “Bitemporal Historisation” anyway?</p> <h2 class="western">Bitemporal Historisation - Overview</h2> <p class="western">Once you ‘get it’, the approach is conceptually very simple. There are essentially just TWO datetime pairs to consider:</p> <p class="western"><b>1 – Transaction datetimes.</b> These from/to datetimes show when the <i>warehouse</i> table is populated. They effectively constitute a ‘version number’ for the data. If we want the latest ‘version’ of data, we query using a ‘high datetime’. If we want the version of data which existed yesterday, we query using yesterday’s date. This datetime-pair provides <i>full auditability</i> of results. Note that 99.999% of the time we will always query using the high datetime (current version, latest transactions). This is a standard SCD type 2 loading process. EVERY table must have these datetimes. The rest of this document will use the term 'Transaction' datetimes, to denote when the record was actually transacted, or committed, to the database.</p> <p class="western"><b>2 – Business datetimes</b>. These from/to datetimes show the period to which the data actually relates. NOT every table will have these datetimes – for many queries we are happy to use the <i>current</i> version of, say, a mappLine truncated
 
 <pre class="western" style="padding-left: 30px;">
 SELECT coverage
@@ -48,7 +48,7 @@ FROM customer_coverage_table AS c
 WHERE c.Contact_LName = 'Fudd'
 AND (c.BusinessFrom le '2020-04-01:00:00:00'dt lt c.BusinessTo)
 AND (c.TransactionFrom le '2020-04-03:00:00:00'dt lt c.TransactionTo);
-</pre> <p class="western">Why aren't we using BETWEEN? Because between is <a href="https://sqlblog.org/2011/10/19/what-do-between-and-the-devil-have-in-common#:~:text=See%20the%20full%20index.,range%20%E2%80%93%20not%20everyone%20gets%20that.">evil</a>!</p> <h2 class="western">Bitemporal Prerequisites and Implications</h2> <p class="western">Implementing a bitemporal approach requires a few principles to be adopted.</p> <h3>Records are Never Modified</h3> <p class="western">With the exception of the TransactionTo datetime field (and maybe the PROCESSED_DTTM in the DDS model), once loaded, a record must <strong>never be modified</strong> (or deleted). This would violate the objective of query repeatability.</p> <h3>Matching Close / Open Dates</h3> When a transaction is closed out and re-opened, or if business values are changing over time, the <em>closing</em> datetime must equal the <em>opening</em> datetime. This is to prevent the "temporal gap" that can happen when you close out a record at, say, 23:59:59 and re-open it at 00:00:00. What happens if you query at "23:59:59.5" ? The data has disappeared!! Note - not all ETL tools have this capability. It's common for an SCD2 load to add a second, or a day, when opening new records. <h3>Business / Transaction FROM must be less than the TO value</h3> Leading on from the previous point, FROM and TO dates cannot be equal, and it also follows that queries should be formed as follows:
+</pre> <p class="western">Why aren't we using BETWEEN? Because between is <a target="_blank" rel="noopener" href="https://sqlblog.org/2011/10/19/what-do-between-and-the-devil-have-in-common#:~:text=See%20the%20full%20index.,range%20%E2%80%93%20not%20everyone%20gets%20that.">evil</a>!</p> <h2 class="western">Bitemporal Prerequisites and Implications</h2> <p class="western">Implementing a bitemporal approach requires a few principles to be adopted.</p> <h3>Records are Never Modified</h3> <p class="western">With the exception of the TransactionTo datetime field (and maybe the PROCESSED_DTTM in the DDS model), once loaded, a record must <strong>never be modified</strong> (or deleted). This would violate the objective of query repeatability.</p> <h3>Matching Close / Open Dates</h3> When a transaction is closed out and re-opened, or if business values are changing over time, the <em>closing</em> datetime must equal the <em>opening</em> datetime. This is to prevent the "temporal gap" that can happen when you close out a record at, say, 23:59:59 and re-open it at 00:00:00. What happens if you query at "23:59:59.5" ? The data has disappeared!! Note - not all ETL tools have this capability. It's common for an SCD2 load to add a second, or a day, when opening new records. <h3>Business / Transaction FROM must be less than the TO value</h3> Leading on from the previous point, FROM and TO dates cannot be equal, and it also follows that queries should be formed as follows:
 <pre class="western" style="padding-left: 30px;">
 SELECT *
 FROM sometable as t