Securing enterprise data is hard. Securing it at enterprise scale is harder still. Most organizations don’t fall short on data masking because they lack tools. They fall short because the way they use those tools can’t keep up with the number of systems that need protecting.
This post looks at why traditional, centralized masking struggles at scale, and how a federated, parallel approach using distributed “worker bees” changes what’s achievable.
The Problem Is Scale
Consider the size of a typical enterprise data estate:
- Hundreds, sometimes thousands, of applications.
- Thousands, sometimes tens of thousands, of database instances across development and test.
- Millions of data points in each, many of them containing personally identifiable information (PII).
Every copy of production data in a non-production environment is a potential breach. Regulations such as GDPR expect personal data to be protected wherever it lives, not only in production.
Why Traditional Masking Is Slow
Even when an organization knows what data needs protecting, which is a big if, the hard part is building the remediation. Whichever masking tool is in use, the traditional process looks like this:
- A specialist team profiles each platform to find sensitive data.
- They design and build masking scripts for that platform.
- They run the scripts, check the results and fix the errors.
- They move on to the next platform.
Each platform typically takes 8 to 12 weeks, and the scripts are often prone to errors and omissions when they’re finally run. The work is usually done by a central team of data experts using a single tool, one platform at a time.
Do the Maths
Imagine a fictional “Small Bank of Narnia” with 100 key platforms. At roughly two months per platform, done one after another:
100 platforms × 2 months ÷ 12 months = about 16 years to reach compliance.
In practice, no organization waits 16 years. Instead, it masks the half-dozen most important systems and hopes auditors and regulators don’t look too closely at the rest. That leaves most of the estate exposed.
Centralization Is the Bottleneck
Scale isn’t the only problem. The bigger issue is that centralized masking can’t be parallelized. One team, one tool and one queue means every platform waits its turn.
Now imagine each of the 100 platform teams could mask their own data. To do that, they would need:
- A way to understand their data without being data experts.
- A reliable way to remediate it without hand-writing scripts.
- The technology to run that work in parallel with everyone else.
With those three things in place, the same program could, optimistically, shrink from 16 years to around six months.
A Different Approach
That observation shaped the design of Enov8 Test Data Manager (TDM). Rather than following the centralized pattern of traditional test data management tools, it was built around four principles.
1. Simple to Use
Platform teams shouldn’t need specialist skills to protect their data. A guided interface takes users through each step, so teams can do the work themselves.
2. Hands-Off
Tasks that were historically manual or semi-manual, such as finding sensitive data and writing masking rules, are automated.
3. Parallel Data Operations
Profiling, masking and validation can run across many platforms at the same time rather than one after another.
4. Enterprise Visibility
Compliance and security teams get a single view of coverage and status across the estate, instead of blind spots.
How It Works
TDM is designed for technical staff, such as engineers and DBAs, and non-technical staff, such as audit and compliance teams. It guides users through a best-practice data security journey:
- Profile. Automated discovery scans your data to find sensitive fields and identify risk.
- Build. Masking rules are generated automatically from the profiling results, so no central team spends weeks writing error-prone scripts.
- Execute. Masking runs on demand or on a schedule.
- Validate. Masked data is automatically tested to confirm it’s compliant and free of PII.
- Report. Dashboards show coverage and status, giving compliance, security and audit teams confidence that risk is going down.
Because masking methods are applied consistently across platforms, the same value is masked the same way everywhere. That keeps end-to-end test data aligned between systems and avoids data-related defects.
Worker Bees: Spreading the Load
For the more technically minded, the key to running at scale is distribution.
Instead of a single masking server processing one database after another, TDM uses lightweight agents called worker bees. Each worker bee carries out data operations close to the data it’s working on. Hundreds of operations can run in parallel, so there’s no need to wait for one application to finish before starting the next.
Worker bees can be placed anywhere on your network, across platforms, subnets, data centers and clouds. Running them close to the data has three practical benefits:
- Speed. Work is spread across many agents instead of queuing behind one.
- Lower latency. Data doesn’t have to travel across the network to a central server and back.
- Lower cost and risk. Less data moves between locations, which cuts transfer costs and keeps sensitive data where it already lives.
TDM also provides a REST API and webhooks, so masking and validation can be built into your existing delivery pipeline. That makes data compliance a standard step in the toolchain rather than a separate project.
Where to Start
Moving from centralized to federated masking doesn’t have to happen all at once. A practical path is:
- Profile the whole estate first. Knowing where sensitive data sits tells you where the real risk is.
- Pilot with two or three platform teams. Choose teams that are keen and whose systems are representative.
- Standardize masking rules. Agree how common data types, such as names, addresses and account numbers, are masked so results stay consistent.
- Roll out in waves. Add teams in parallel rather than one at a time, and track coverage on a shared dashboard.
To Summarize
In the past, a single team of specialists could mask data for a handful of platforms, and for a small estate that may still work. But today’s IT and test environments are far more complex. Even medium-sized organizations run hundreds of data platforms holding gigabytes or terabytes of data.
If your organization wants to be truly compliant, rather than compliant on paper, centralized and sequential masking won’t get you there. It’s time to automate, federate and parallelize your data operations.
Learn more about Enov8 Test Data Manager or talk to our team.
Jane Temov
Jane is an experienced IT Environments Management & Data Evangelist. Areas of specialism include IT & Test Environment Management, Data Securitization, Release Management, Service Resilience, Configuration Management, DevOps & Infra/Cloud Migration.