Database Settings
Database settings define parameters specific to the database connection that currently has focus in the project, i.e. the database node that is effectively selected in the Project explorer tree.
These settings are saved to the project file.
When a project file is opened the settings from the project file are loaded into memory which can then be accessed from the Settings>Database tab in the main GUI window.
Under the Database tab are settings that apply only to the currently selected database.
Oracle users please note that whenever the term "database" is used within DataVeil it is actually referring to an Oracle schema, unless otherwise noted.
The Connection Tab
The connection details and credentials are specified in the Connection tab.
An example for an SQL Server connection is shown below. Please note that the fields shown in the Connection tab shall vary according to the database type.
Please refer to the Connection section for further details.
Execution parameters such as maximum thread count and batch sizes are specified here.
Masking Connections
This represents the maximum number of long-running threads that shall be running concurrently.
This applies particularly to masked value generation threads (each mask uses a dedicated thread) and the number of threads used to write masked values to tables.
Occasionally this thread count may be exceeded for very brief periods by short-running threads.
If you find that your environment is facing resource pressures then reducing this value shall reduce resource contention.
If you have adequate resources then increasing this value may reduce the overall execution time of your masking project because more tasks shall be run concurrently.
Masked Value Generation
This represents the maximum number of masked values that are generated and written to mapping tables in a single SQL query.
Higher values increase throughput and reduce overall execution time but may also increase undo log space held for the duration of the query.
The default value is 800,000. Although this number may seem high it does not cause problems in most modern environments.
Table IO
NEW
Using NEW, DataVeil shall create a copy of the original table except that it shall replace the sensitive columns with columns containing masked values.
DataVeil shall then create all the constraints, indexes and triggers that were found in the original table.
If all of this is successful then the original table is dropped and the copy shall be renamed to the name of the original table.
Considerations:
* Please ensure that you have sufficient table space to accommodate duplicate tables of all those tables being written using the NEW table IO method.
* For Oracle, writing to tables using this method is much faster than UPDATE and it significantly reduces writes to the redo log.
* The NEW method can still be slower overall than UPDATE because if the original table contained a great number of indexes and constraints (including 'NOT NULL' constraints) then each of these must be created in the new table, whereas when using UPDATE only those indexes and constraints that involved masked columns must be created in the new table. This can be a significant workload. For example, suppose you have a table with 20 indexes and 50 constraints and you are only masking 1 column that affects 1 index and 1 constraint. If you use NEW table IO then all 20 indexes and 50 constraints shall need to be created that can take a very significant amount of time. If you use UPDATE table IO then only 1 index and 1 constraint shall need to be created.
* This setting is not available for Azure SQL DB, MySQL or SQL Server.
UPDATE
Using UPDATE, DataVeil shall overwrite only the sensitive columns in the actual original table.
DataVeil only needs to rebuild indexes and constraints that are dependent on the masked columns. All other original indexes and constraints will remain unaffected.
Considerations:
* For Oracle, if DataVeil needs to rebuild an index and the table has statistics locked then DataVeil shall unlock statistics on the table to allow the index to be rebuilt. DataVeil shall leave statistics unlocked for tables that had indexes rebuilt. All other table statistics locks shall remain unchanged.
Setting an Explicit Table IO for an Individual Table
You can override the default table IO method, described above, on a per-table basis.
To do so, select the table node in the Project explorer tree and then open the Masking -> Execution tab for that table.
The default is DEFAULT which means that the table IO method defined for the Database (as described above for Settings -> Database -Execution) shall be used for this table.
To override the table IO for this table simply choose one of the other settings (UPDATE has been selected in the example screen capture above).
Upon Run Completion
Restore
Upon a masking run completion, the state of each index, constraint and trigger that DataVeil manipulated during the masking run shall be restored to the same enabled or disabled state that each was in at the start of the masking project execution.
Enabled
Upon a masking run completion, the state of each index, constraint and trigger that DataVeil manipulated during the masking run shall be set to the enabled state regardless of the state that each item was in at the start of the masking project execution.
Native functions are only available for Oracle and SQL Server databases. This tab shall not be shown for other database types.
Within DataVeil, the terms 'native functions' and 'native library' refer to DataVeil-supplied modules that can be installed on a DBMS to perform the masking function instead of the default DataVeil SQL functions.
For Oracle, the module is a Java JAR and for SQL Server the module is a CLR Assembly. Please refer to DataVeil Native Library in the Installation section for further information.
Use DataVeil native functions if available on the DBMS
If this option is selected and the native library is installed on the DBMS then DataVeil shall call those functions instead of SQL functions, wherever possible, to generate masked values. The benefit is significantly faster performance.
If this option is not selected then DataVeil shall only use its built-in SQL functions.
If not available then log:
Warning
Enabling this option will cause compile and runtime warnings to be logged if DataVeil is unable to use its native functions. Execution of a masking project shall continue and DataVeil shall use only the built-in SQL functions.
Error
Enabling this option will cause compile and runtime errors to be be logged if DataVeil is unable to use its native functions. Execution of a masking project shall not proceed.
Function call prefix:
Each function call shall be qualified with this prefix.
The main benefit is that this setting enables you to install the DataVeil native library to a location different to the default and even share one installation of the native library rather than installing these on every database being masked.
Please refer to DataVeil Native Library in the Installation section for further information.
The DBMS Tab
The DBMS tab shall only appear for DBMS's that have any such settings. Currently, this is only applicable for SQL Server.
SQL Server
Require Simple recovery model
Ordinarily, DataVeil attempts to change the recovery model to Simple if it is not already.
It is recommended that this setting is enabled. The consequences of Simple recovery model not being used are:
* Transaction log shall continue to grow whereas Simple shall truncate the log after each transaction.
* Sensitive data shall be present in the transaction log potentially enabling someone to recover the original sensitive data.
If this option is enabled then DataVeil shall require that the recovery model is either already Simple or that DataVeil is able to temporarily change the recovery model to Simple. Otherwise DataVeil shall log an error and terminate the masking run.
In some particular cases, users may wish to proceed with a masking run even if the recovery model cannot be changed. Therefore, by disabling this option DataVeil shall allow the masking run to proceed even if the recovery model is not Simple.
If system versioned temporal table is masked and history is not masked then truncate history
A system versioned temporal table has a corresponding history table that contains previous values of the temporal table.
If a temporal table is masked, meaning that it contains sensitive data, then it is very likely that the history table also contains sensitive data.
Therefore, if the user has configured masks on the temporal table but not on the history table then DataVeil will truncate the history table if this setting is enabled. If this setting is not enabled and no masks have been defined on the history table then a compile error will be reported.
In case there is a need to mask a temporal table but preserve its history table then configure a Preserve mask on any column in the history table. This will prevent it from being truncated or causing the compile error mentioned.
When truncation is to be performed, it will not be performed during preview runs and therefore not apparent in the data browser although a log message shall be written to indicate that the table would be truncated in a masking run. In an actual masking run the truncation will be performed and a confirmation message shall be written to the logger similar to "Truncating history table [dbo].[my_hist_tbl] of sys versioned table [dbo].[my_tbl]".
If using DataVeil native library and slowness detected then clear SQL Server system cache on masking start
Occasionally, the performance of the DataVeil native library may appear to be poor.
For example, performing the simple version query SELECT DataVeilNativeVersion() may take several seconds when in fact it should take only milliseconds. This may be due to a caching issue on SQL Server. Clearing the cache resolves the slowness.
Enabling this option means that upon a masking run start, DataVeil will check the response time of the DataVeil native library and if it is detected to be slow then DataVeil shall automatically clear the SQL Server system cache.
Take snapshot of original values...
After DataVeil completes generating all masked values and immediately prior to overwriting the original values, DataVeil shall make a temporary backup copy of only those original values to be overwritten if this checkbox option is selected.
Therefore, if an error occurs during the write phase then DataVeil shall attempt to restore the original values from its temporary backup copy ("snapshot") so that the tables can be in a consistent state. DataVeil purges this temporary backup copy upon run completion.
If this option is disabled and an error occurs then DataVeil shall not be able to restore the original values which may leave data in an inconsistent state. Therefore, it may be necessary to perform a DBMS database restore to regain a consistent database.
Ordinarily it is recommended for this option to be enabled (the default). This is because the backup copy is usually very quick and may save you additional time and effort in avoiding a manual DBMS restore if something goes wrong. E.g. One of the most common causes of a write phase fail is if the user inadvertently configures a mask that generates duplicate values for a column that has a unique constraint. In this case a constraint violation error shall occur after all masked data has been written and an attempt is made to rebuild the unique constraint. Note: A Complete Preview will detect such problems whereas a Quick Preview will not.
This option can be useful in those situations where this snapshot actually does take a considerable length of time and you are very confident that the write phase shall be successful. The snapshot can take a long time when masking very large tables and especially if the tables contain very wide columns that are being masked. The effect is that there will be a significant delay between when all masks have completed and before the Table Writer tasks are started - sometimes users believe that the masking run has hung whereas it is actually just the snapshot process taking a long time. You can enable the logging of diagnostic messages if you wish to see more detailed masking run execution information.