DevOps that Data! You will never have to worry about getting realistic databases for dev, test and CICD again.
vME, pronounced /’vi:mi/, a member of Enov8’s “Test Data Management Suite”, is a database provisioning technology using “cloud native” containerization & cloning (database virtualization) methods. A solution that lets you “economically” spin up dev, test environments in seconds & circumvents traditionally slow data operations & storage issues.
The Data Challenge
Enterprise Data Management can often be challenging due to its complexity, size, and sophistication, leading to issues such as long processing times, high costs, low data quality, difficult serviceability, and inadequate security.
- Cadence:The deployment of test data and environments can take anywhere from weeks to months, slowing down software development and analytics projects. This can be especially problematic for urgent Business as Usual endeavors.
- Cost: Data Provisioning can be expensive due to the large amount of data that applications require. Data sources of tens of terabytes may require high-performance storage and disks, and specialized professionals may need to be accessed, leading to further costs.
- Quality: Data Inconsistencies and Inaccuracies can occur when data is stored in multiple, disconnected silos, is prepared in inconsistent or non-standardised ways, or is allowed to become outdated. This can have negative effects on Analytical, Engineering, and Quality efforts.
- Serviceability. Development and Test teams often lack the control over their data, causing them to have to rely on specialized DBAs or Data SMEs to meet their needs. This often leads to delays in the process, as these experts have numerous other tasks to attend to. This can also inhibit their ability to rollback, refresh, and retest.
- Security: Organizations must protect their sensitive data and comply with data privacy regulations. Without adequate data security measures, organizations can risk data breaches and non-compliance penalties.
All of which adds up when you consider that every application needs between five and ten test environments, such as System Test, System Integration, User Acceptance, Staging, and Production Support. Plus dedicated environments for users, tribes and project teams.
vME is a database provisioning tool that lets you create full copies of your data & databases in seconds, using around 40 MB of disk space per clone.
Instead of spending hours provisioning multiple copies of your database for development, testing, or diagnostics, vME ingests a single ‘image’ of a database, which is used as the source for multiple clones. Each clone works just like a normal database, but takes only a few seconds to create and requires only around 40MB of disk space.
With vME’s web app and built-in API and CLI (Command Line Interface), engineers spend much less time on provisioning requests. Meanwhile, developers and testers can work on up-to-date, isolated copies of the database to speed up development, accurately test code, and fix issues faster.
Database virtualization technology enables the cloning and sharing of dataacross different environments. A method that reduces the total footprintby x10 and speed by x100.
*Enov8 TDM, which integrates with vME, also provides data masking and masking for sensitive data for compliance and security, so organizations can protect their data from unauthorized access.
Under the Hood
Under the hood, Enov8 vME is an open virtual appliance* (OVA) designed to simplify the DataOps process and virtualize database infrastructure.
*Think of an appliance as a plug & play toaster.
You can employ one appliance, or you can horizontally scale, by having multiple appliances spread across your domain, clouds or teams.
vME DataOps Platform is built by combining two key technologies into a single unified DataOps usecase. The first technology used is containerisation. Through the use of containers, vME provides an efficient and consistent environment for databases, reducing space requirements, improving agility,manageability, and performance. This results in faster and more reliable datadeployment compared to traditional virtual or physical machines. The second technology used is the Zettabyte File System (ZFS), an advanced volume manager that allows us capabilities like data compression, data pooling, point-in-time snapshotting and clones.
Enov8 vME workflow can be described in 4 simple steps:
- Ingest you Data Source
- Snaphshot your Data Source *Point in Time
- Clone the Snapshot
- Provision & Use the Clone
Benefits of vME
Some of the key benefits of vME include:
- Accelerating environments & data deployment. vME allows you to deploy databases in seconds.
- Reducing storage requirements. vME databases are tiny (in MBs not TBs).
- Promoting Data DevOps as code. Codifieable & DevOps-able through easy touse CLI & APIs.
- Preventing Data Loss. The vME architecture ensures data stays on-premise orinside your private clouds.
- Better environment housekeeping. Easily provision & decommission unused environments.
- Improved productivity. No disruption! Spin up databases to support individual or team activities.
- High Scalability. Appliance(s) are designed to be federated. Meaning vME can be installed across your network and divisions for highest perfomance.
- Simplified DB lifecycle management. Automate the creation, storage, and deployment of database clones, making it easier to manage the database lifecycle.
- Promotes data privacy and security. Control who has access to your data by automatically encrypting and masking the data through Enov8 TDM integration.
Supported Data Platforms
Supported Data Platforms
Our vME appliance manages the database servers through containers and the underlying data at the file system block level. Simply put, that means our underlying engine is 100% database or file type agnostic.
vME supports all the most popular database types including AWS RDS, Azure Databases, Couchbase, DB2, MariaDB, Microsoft SQL, MongoDB, MySQL, Oracle, Sybase, Postgres, Files & Directories. And can be easily configured for other data types.
The Enov8 vME DataOps Workflow:
1. Data Ingestion
In Enov8 vME, data ingestion refers to the process of bringing data into the Enov8 vME platform from various sources, such as databases, filesystems, and cloud environments.
This process involves extracting a copy of the data from the data source (for example a backup-file or database.SQL) and loading (ingesting) it into the Enov8 vME platform. The ingested data is then used to create virtual copies, which can be provisioned to development, testing, and analytics teams.
During the data ingestion process, Enov8 vME can also performs various tasks on the data, for example: Data Profiling, Data Masking or Encryption, Compliance Validation, Fabrication through the Enov8 TDM Engine.
3. Data Cloning
In vME, Data Cloning in is a process of creating an exact ‘point-in-time’ copy of an existing datasource (typically a database or set of files) that has undergone snaphotting.
Each clone is an exact copy of the source database, but it is stored in a smaller, lighter-weight format. This makes it much faster and easier to deploy and manage multiple copies of the same database. From all intensive purposes a clone sometimes refered to as a Virtual Database (VDB) is a fully functioning copy of the dataset and is ready for provisioning and consumption.
2. Data Snapshotting
In Enov8 vME, data snapshotting is the process of taking a point-in-time copy of source data ingested into the vME appliance. This snapshot is subsequently used to create a virtual copy of the data, which in vME is called a clone.
Data snapshotting in Enov8 vME allows organizations to create multiple virtual copies of the same data, each with a different point-in-time snapshot.
This allows teams to work with the most up-to-date and relevant data, while reducing the time and resources required to provision and manage data. It also enables teams to roll back to a previous snapshot, if necessary, which can be very useful in case of data corruption or accidental changes.
Even if you have enough disk space to accommodate multiple database copies, traditional provisioning processes are fundamentally too slow to support the team’s requirements.
With vME, you can copy a database, TB is size, in just seconds. This rapid provisioning allows individuals to work on their own isolated copy of the database, test in parallel and experiment while being able to refresh clones in seconds, on demand.
Tip! Using vME’s native automation, snapshots and clones can be created quickly and easily, for example during a night-time maintenance window.
In addition to vME offering a light and intuitive web interface for easy administration of your data. vME also provides a complete API and Command Line Interface (CLI) for automation of your data operations.
Easy to learn, the Enov8 API & CLI can be easily included in your favourite scripting language or CICD* tool. Thus supporting displacement of manual data operations and allowing task scheduling.
*vME integrates easily with all CICD tools, including: Jenkins, Travis CI, CircleCI, GitLab CI/CD, Azure DevOps, AWS CodeDeploy, Buddy, Bitbucket Pipelines, GitLab CI/CD, and CodeShip.
Why Enov8 vME?
A few of the key benefits found in uplifting your DataOps capability.
Our solution is easy to learn, painless to on-board, has a straightforward “end to end” test data management process and is both tech & non-tech friendly.
Use our platform to support compliance objectives. Identify risk, avert a security breach, protect customer identity and avoid huge penalties.
Increase DevSecOps velocity through fast risk analytics, rapid risk remediation, automated compliance validation & push-button consumption of data.
Our vME solution supports continuous data integration, continuous data delivery, automate-able operations and promotes team standardization & repeatability.
Managing test environments and test data preparation can be just as hard as managing your Production Environments. Keep up to date with the latest resources, reports & whitepapers to grow your knowledge of understanding of groundbreaking tools to accelerate digital transformation.
Blogs & Updates
Stay in touch with Enov8 and our TEM / TDM Network. Receive regular updates and insights from the industry, and find out what we’re working on! We love to contribute to optimizing your software development environment, testing environment, and production environment.
Kick Start Your Digital Transformation
Contact us to get started today!