Parallel Masking and Intra-Column Parallelism
DataVeil supports parallel masking so that multiple masking operations can be performed at the same time.
In DataVeil versions prior to 5.1, this parallelism operated on the basis of one masking thread per column.
Since DataVeil 5.1, DataVeil also supports intra-column parallelism. This feature allows a single eligible column to be processed by more than one masking thread at the same time.
Intra-column parallelism can improve performance when a project contains one or a small number of large or long-running columns. Without intra-column parallelism, other masking threads may finish their assigned columns and then remain idle while the project waits for the remaining long-running column to complete. With intra-column parallelism, DataVeil may divide eligible work for that column across multiple threads.
Intra-column parallelism does not change the masking rules configured in the project. It only changes how eligible masking work is divided and executed.
When Intra-Column Parallelism Is Used
Intra-column parallelism is automatically enabled for new and existing projects.
DataVeil determines at run time whether intra-column parallelism can be used for a particular mask. It is used only when all required conditions are satisfied, including database platform support and mask eligibility.
If intra-column parallelism cannot be used for a mask, DataVeil will process that mask without intra-column parallelism.
Intra-column parallelism is most useful when masking time is dominated by one or a few large columns. It may provide little or no improvement when a project already contains many columns that are being processed concurrently, or when the main performance bottleneck is database writing, logging, storage I/O, recovery snapshot creation, index creation, or another DBMS operation.
Disabling Intra-Column Parallelism
Intra-column parallelism is enabled by default and is the recommended setting.
It can be disabled at Settings->Database->Execution
Disabling intra-column parallelism does not disable DataVeil’s normal parallel masking. It only prevents a single column from being processed by multiple masking threads.
Identifying Intra-Column Parallelism
DataVeil log messages indicate whether intra-column parallelism is being used.
The GUI Execution Control panel also identifies masks using intra-column parallelism. When intra-column parallelism is being used, the Code column includes a .P suffix.
For example:
* SQL.P
* Native.P
The .P suffix means that the mask is being processed using intra-column parallelism.
If the suffix is not shown it simply means that intra-column parallelism was not applicable for that mask, database platform, preview mode, or configuration.
Supported Database Platforms
Intra-column parallelism relies on database partitioning support.
It is available for database platforms and editions that provide the required partitioning support, including:
* SQL Server
* Azure SQL Database
* Azure SQL Managed Instance
* Oracle
* PostgreSQL
Intra-column parallelism is not available for MySQL.
If the connected database platform or edition does not support partitioning then DataVeil will process masks without intra-column parallelism.
Mask Eligibility and Limitations
Not all masks and mask configurations are eligible for intra-column parallelism.
Intra-column parallelism is not used for:
* Shuffle masks
* Number Sequence masks
* Columns using the Column Duplicates option
* Quick Preview
The presence of a mask that is not eligible for intra-column parallelism does not prevent other eligible masks from using it. This also applies where eligible and non-eligible masks are used within the same column.
Performance Considerations
Intra-column parallelism uses available masking threads defined by the Maximum Mapping Threads setting.
It is generally recommended that this value be set at 1x to 2x the number of physical cores that will be available to DataVeil and shall not be significantly busy with other tasks during masking.
Increasing the Maximum Mapping Threads value may improve performance in some projects, but setting it too high can cause contention thereby reducing overall performance and throughput. The best value depends on the database platform, server resources, storage performance, network latency, mask types, and project configuration.
The Mapping Batch Size setting may also affect intra-column parallelism. In general, a recommended setting is 100,000.
The optimal values for Maximum Mapping Threads and Mapping Batch Size are project-specific and it is encouraged to perform benchmark tests in the user's environment to determine these optimal values. The default values provide a very good starting point.