Holistic Test Data Management – Beyond ETL
by Niall Crawford
Your project puts in a request, they grab copies from production, mask them, subset them and deploy them into your Test Environments. Easy Peasy! … Well probably not, probably a lot of paperwork, engineering & provisioning effort. And then the issues really start.
- The data lacks end-to-end integrity i.e. the data is broken.
- The developers & testers can’t easily find the data they are looking for.
- And when the IT teams do find the correct data-points they can use, they all use it.
Causing contention & data related test defects. And it all starts to grind to halt, suddenly Development & Test cycles are being blown out.` And then to add insult to injury, an honest Test Analyst, notices that not all the data has been masked. A serious concern when one considers that’s where your project teams spend 95% of their time, and the opportunity for information to be misplaced or stolen is high. A suboptimal situation, which exposes the customer to:
- Identity Theft & Fraud
And exposes your own organisation to
- Compliance Penalties
- Industry sanctions
- Brand Damage
- Consequent Lawsuits
Not exactly ideal, particularly with data compliance legislation like GDPR that will sting you for 4% of turn-over. Yet I can virtually guarantee, sadly, that the above scenarios describe most organizations today. The reason why data is such a problem – is sixfold:
- Enterprise architectures are typically diverse & distributed.
- Environment & data footprints are under constant change.
- Individual databases are often large, poorly defined or understood.
- System & data documentation often suffers from technical debt.
- Easy to make mistakes during data Subsetting (causing integrity health issues).
- Very easy to make mistakes during data obfuscation exercises (causing PII leakage).
The traditional ETL approach to Test Data Management simply isn’t good enough.
- It is too Slow
- It is too Manual
- It is too Error-Prone &
- It is not customer / user-centric
There is a fundamental need to recognise that successful Test Data Management can’t rely on ETL alone. Instead, organizations must start looking at data a little more broadly and leverage more automation to ensure accuracy, quality, compliance & ease of end-user consumption.
Enter the Holistic Test Data Management (HTDM) Framework
Designed by Enov8, the HTDM Framework is used to contextualise the broader aspects of Test Data Management. A set of “Lego” blocks that call out the broader considerations and needs of an automated test data solution. Build around the traditional ETL, the HTDM promotes supporting capabilities like: Data Requirements Capture – So you have a clear understanding of consumers (testers & projects) needs. Automated Data Profiling – To rapidly understand data structures and PII risks (pre-ETL). Automated Data Validation – To rapidly determine if created data (post-ETL) is free of production patterns and healthy (has integrity). Test Data Mining – So Testers can visualise, understand and find end-to-end (cross-system) data without building complex queries and scripts. Test Data Bookings – So test data can be assigned to test cases or teams and avoid the risk of overwrite.
Key Benefits of creating an HTDM Framework
- Understand your Data
- Improve Compliance.
- Ensure Data Health
- Ease of Consumption (based on your needs, healthy & easy to find)
All of which leads to happy testers and streamlined project delivery.
Learn More or Share Ideas
If you’d like to learn more about Data, Release or Environment Management or perhaps just share your own ideas then feel free to contact the enov8 team. Enov8 provides a complete platform for addressing organisations “DevOps at Scale” requirements. Providing advanced “out of the box” Holistic Test Data Management, IT & Test Environment Management & Release Management capabilities.
Innovate with Enov8, the IT Environment & Data Company.
Specializing in the Governance, Operation & Orchestration of your IT systems and data.
Delivering outcomes like
- Improved visibility of your IT Fabric,
- Streamlined Delivery of IT Projects,
- Operational Standardization,
- Security & Availability,
- DevOps / DataOps Automation,
- Real-Time insights supporting decision making & continuous optimization
Niall is the Co-Founder and CIO of Enov8. He has 25 years of experience working across the IT industry from Software Engineering, Architecture, IT & Test Environment Management and Executive Leadership. Niall has worked with, and advised, many global organisations covering verticals like Banking, Defence, Telecom and Information Technology Services.
15JULY, 2021 by Justin ReynoldsCompanies go to great lengths to protect their physical environments, using deterrents like locks, fences, and cameras to ward off intruders. Yet this same logic doesn’t always translate to digital security. Corporate networks — which...
06JULY, 2021 by Justin ReynoldsCompanies today face increasing challenges around reducing the time and cost of software development. Many are thus using DevOps methodologies, which combine software development and IT operations to achieve continuous delivery and...
24JUNE, 2021 by Omkar HiremathInformation technology and the digital world don’t exist without data. The data of an organization can contain a lot of unclassified, as well as classified information. Irrespective of that, only authorized personnel should have access to...
14JUNE, 2021 by Kathrin PaschenManaging releases at scale is daunting. It involves juggling dependencies, timelines, and requirements. The stakes can be pretty high, too. Not all failures are as spectacular as crashing a lunar lander or losing $440 million. Even so,...
28MAY, 2021 by Sasmito AdibowoThe benefits of using cloud environments to store and access data over the Internet has been highly beneficial for many businesses. Cloud environments help both start-ups and enterprises scale up conveniently. However, as with other major...
10MAY, 2021 by Eric GoebelbeckerImagine a technology that lets you focus on your business logic and that takes care of issues like reliability and scaling for you. What would it be like if you only had to pay for the computing time you use rather than pay by the day,...