--- title: 'Horror Stories of Business Data Ingestion' description: Real-world tales of business data ingestion gone wrong - from broken ETL jobs to network-share "audit trails" and the Access database under the operations desk. date: '2026-08-15 09:00:00' author: 'Data Controller' authorLink: https://www.linkedin.com/showcase/data-controller-for-sas previewImg: './horror.jpeg' tags: - Master Data Management --- # Horror Stories of Business Data Ingestion > "We spent 6 weeks recruiting a Data Engineer, who built several complex ETL workflows, which promptly broke on his departure thanks to a minor change in the source file" > "I wrote the SQL update script myself, which was working fine until my holiday cover adjusted the where clause, and wiped half the records. At least, that's what I think happened. We didn't see the discrepancy until the following month." > "We often need to upload new mapping files and so all the Business Analysts in the Customer Insights team are being trained on the enterprise ETL tool" > "Our DBA does the manual updates, but he only has 0.5 free days per month allocated to the task - and he will only do it AFTER month end" > "We have a dedicated data engineering team and they can knock out an ETL job in a few minutes. The problem is that it takes 4-8 hours of documentation and meetings to set the requirements, and 4-8 weeks to deploy the job into production" > "If you need to change the data values, just double click the table in our .NET application and type it in. Remember to close it afterwards though, else the overnight batch job will fail due to a physical lock. This happens quite a lot - it's safest to just close the application afterwards and re-open it." > "We keep an audit trail of ALL manual updates. Whoever makes the change will just open the spreadsheet, fill in the necessary records, and save the file with a new version number on the department network share" > "We don't have any business data ingestion, we only load source data, and it is never modified" > "We export the data from the web report into Excel. We copy/paste the data into another Excel sheet where we execute some VBscript macros that were developed by a guy who left the company. After that, the updated data are saved as a csv file on a network drive. The file is then picked up by a batch job and loaded into the database. The job is scheduled to run every 15 minutes. The updated records can then be displayed in our reports." - [Xavier Bizoux](https://www.linkedin.com/in/xavierbizoux) ([source](https://www.linkedin.com/feed/update/urn:li:activity:7183406509450547200?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7183406509450547200%2C7184815374624772096%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287184815374624772096%2Curn%3Ali%3Aactivity%3A7183406509450547200%29)) Coupled with: > "We don't load our data to the DataWarehouse, it's too difficult. We just stick it all in the team ACCESS database on the PC under the operations desk" What are YOUR horror stories?