Command Line Execution

In the delivered software folder you will find the subfolder called batch that contains:

dataveil_cmd_nix – Linux / Unix sample shell script

dataveil_cmd_win.bat – Windows sample batch file

Each sample file shows how to run a DataVeil masking project from the command line and check the completion status after the DataVeil masking job completes so that you can perform other OS commands according to whether the completion was successful or not.

It is recommended that you make a copy of the supplied file that corresponds to your environment to customize for your specific needs.

 

Linux / Unix
  

The syntax to run a DataVeil masking project from a Linux/Unix shell script is shown below. Please refer to the supplied file dataveil_cmd_nix.

This command is on one logical line although it appears on multiple lines below due to line wrap.

./dataveil --nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m --project="/export/home/test/Documents/dataveil/Demo v1/Oracle/kona_sol1.dvp" --key="123456" --refreshschema=true --compilewarning=continue --createdirs=true --log="mybatchlogs/mylog.txt!" --logxml="mybatchlogs/mylog.xml!"

A description of all options specified in dataveil_params is shown in the section "Options" below.

Example:

../bin/dataveil --nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m --project="/export/home/test/Documents/dataveil/Demo v1/Oracle/kona_sol1.dvp" --key="123456" --refreshschema=true --compilewarning=continue --createdirs=true --log="mybatchlogs/mylog.txt!" --logxml="mybatchlogs/mylog.xml!"'

Important for Linux / Unix

The paramaters "--nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m" MUST be included otherwise DataVeil may not run properly in command-line mode.

The entire command should be on one logical line.

You could use a shell variable to contain the parameters but not if you are using the --con parameter with curly braces {} because these braces will interfere with the shell variable's expansion.

Please ensure that the script file has execution privilege.

E.g.  chmod +x dataveil_cmd_nix 

 

Windows
  

The syntax to run DataVeil as a batch process from the Windows command-line is: 

START "MyWindowTitle" /WAIT "dataveil64.exe" ^
 --nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m ^
 --project="<masking-project-file>" ^
 <options>
 
 

Exit codes

0 = Success. No warnings and no errors.
 1 = Success with warnings.
 2 = Errors encountered.
 Note: Other exit codes may be returned from other components such as the underlying environment.
 

Example:

START "MyWindowTitle" /WAIT "..\bin\dataveil64.exe" ^
 --nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m ^
 --project="C:\\Users\\Gen\\Documents\\DataVeil\\Demo v1\\kona.dvp" ^
 --key="123456" ^
 --refreshschema=true ^
 --compilewarning=continue ^
 --createdirs=true ^
 --log="mybatchlogs\\mylog.txt!" ^
 --logxml="mybatchlogs\\mylog.xml!"
  

Important for Windows
  

The line "--nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms64m -J-Xmx512m" MUST be included otherwise DataVeil may not run properly in command-line mode.

The command above shows "MyWindowTitle" after START. According to Microsoft documentation this specifies the title to display in the Command Prompt window title bar. You can also specify an empty string as "" if you prefer. This value MUST be included in order to avoid Windows command line parsing problems despite Microsoft documentation indicating that this value is optional.

The command shown above runs dataveil64.exe for 64 bit Java. If you need to run 32 bit Java then use dataveil.exe instead. Make sure that you specify the correct path to this executable file on your system. The example above shows the relative path from the sample bat file within the delivered software folder structure.

Remember to include the 2 character line continuation " ^" at the end of each line except the last. Note that a space is required before the chevron. These must be the last characters on the line. If any other characters follow the chevron (including whitespace) then this may cause an error with an ambiguous error message.

If there is an error with the START command, such as a syntax error, then the error message shall be written to the current error stream. By default, this is the console window where you run the START command. These errors are not written to the DataVeil logger locations (defined by parameters --log and --logxml) because DataVeil may not yet have been successfully started.

 

Important for All Environments
  

Do NOT run more than one instance of DataVeil concurrently for the same operating system User. DataVeil relies on cached data in a User directory. This does not allow more than one instance of DataVeil from running concurrently for the same User. Multiple DataVeil installations for the same User will not overcome this limitation as all of the User's DataVeil installations will attempt to use the same DataVeil user directory. As the command syntax shows, a /WAIT parameter is specified. This is to ensure that the DataVeil instance completes before resuming the batch file execution. i.e. You can have sequential executions of multiple DataVeil projects in a batch file (but not concurrent).

 

Options
  

The following options can be specified with the command.

--compilewarning={stop|continue}
  

Specifies whether DataVeil should proceed to execute the masking project if compile warnings are encountered. This does not refer to warnings that may be encountered during the actual masking process (i.e. warnings after the compile phase).

The default is stop.

 

--con Connection Substitution

 
 This parameter specifies connection parameters that are to override connection parameters defined in the masking project file.

There are two variations on the syntax: Simple and Complete.

Note: In this context of DataVeil, references to "database" should be interpreted by Oracle users as referring to "schema".
  

Simple Format - Replace only database name
  

If you want to replace only the database name in a project that contains exactly one database definition then you can use this simple form of --con specification.

--con=<new-database-name>

Examples:

--con=stagedb1

--con="db os3"    // Double quotes are required here because the data location name has an embedded space.
    

Complete Format - Replace one or more connection parameters in one or more data location connection definitions


 --con="{<con-param-list>]}[ ,{<con-param-list>]}"

 

* Note that double quotes are required around the entire --con value specification. i.e. One before the first { and one after the final }.

* Each data location's replacement parameters are specified within { and }. If a project contains more than one data location definition then you only need to specify those data locations whose parameters you wish to change.

* If your project contains only one data location then the prevname parameter is optional.

* All spaces within the quoted string are significant. Do not use any quotes within the entire --con value even if even a parameter value contains embedded spaces.

Example:

Note that the newname parameter 'stage db2' does not have quotes around it within the --con value even though it contains an embedded space. This because all spaces within the --con option value are significant.

--con="{prevname=db1,newname=stagedb1,host=q2.test.lan},{prevname=db2,newname=stage db2,threads=14}" 

 

<con-param-list>
  

This specifies one or more <con-param> parameters for a single connection definition.

This will override the corresponding connection parameters defined in the project file for the duration of the execution only. The project file will remain unchanged.

If a connection parameter is not specified then the value defined in the project file shall be used.

*** Never enclose a <con-param> value in quotes. All spaces are significant.

  

newname=<new-schema-name>

This option enables you to mask a different database schema than the one that is defined in the masking project file.

Example: --con="{newname=myotherschema}"
 
 

prevname=<schema-name-in-proj-file>

If multiple database schemas are defined in a masking project file then use the prevname option to resolve ambiguity.

Example: --con="{prevname=origschema1,newname=myotherschema1},{prevname=origschema2,newname=myotherschema2}"
 
 

auth={windows|sqlserver}

Applicable for SQL Server connections only.

Example: --con="{prevname=db1,newname=stagedb1,auth=windows}"
  

code={ native_error | native_warn | native_sql | sql }

native_error: DataVeil native library is to be used. If it is not found or not accessible then a compile error shall be emitted causing DataVeil to terminate with error.

native_warn: DataVeil native library is to be used if possible. If it is not found or not accessible then a compile warning shall be emitted. If the --compilewarning option is continue then DataVeil shall continue execution using DataVeil's built-in SQL masking functions. Note: This is a compile-time warning and therefore shall not cause an execution completion warning to be returned. i.e. If there are no other errors or warnings during masking execution then the completion code shall be 0. If the --compilewarning option is stop and the DataVeil native library is not found then a compile warning shall be emitted and then DataVeil shall terminate with an error code. Please refer to the --compilewarning option.

native_sql: DataVeil native library is to be used if possible. If it is not found or not accessible then DataVeil shall continue execution using DataVeil's built-in SQL masking functions.

sql: DataVeil shall perform data masking using its built-in SQL masking functions. If a DataVeil native library is available it shall not be used.

The default is native_sql.

Example: --con="{code=native_error}"
 
 

dbms={ azuresql | mysql | oracle | sqlserver }

Specifying the dbms parameter is only necessary if it is different to the connection's original dbms type that is configured in the project file.

Specifying a different dbms type will cause a Migrate Masks to be performed for the masks that are configured for this connection in the project file.

It is important to specify the new connection parameters required for the the new connection and in the manner they are expected. For instance, Oracle schema names are typically case sensitive and are often expected in upper-case. The example below shows that the newname parameter has been specified in upper-case otherwise the Oracle connection could return an error message indicating that the schema does not exist or that the user does not have access.

Example: --con="{newname=STAGEDB1,dbms=oracle,servicename=pdborcl,port=1521,user=test01,password=123456}"
  

host=<value>

<value> can be a host name or an IP address.

Example: --con="{prevname=db1,newname=stagedb1,host=srv1.test.lan}"
 Example: --con="{prevname=db1,newname=stagedb1,host=123.12.3.1}"
 
 

instance=<value>

Applicable for SQL Server connections only.

<value> is an SQL Server instance name.

Example: --con="{prevname=db1,newname=stagedb1,instance=INST03}"
  

jdbc=<value>    (for Oracle)

<value> is a complete JDBC connection string. If this value is specified then do not specify parameters servicename, sid, tnsname, host and port because they shall be expected to be contained within the JDBC string value. You should specify the user and password either as separate parameters (above) or within the JDBC string value (not in both places). If the project file already defines user and password values then those values shall be assumed unless you override them with the user and password <con-param> parameters.
  

jdbc=<value>    (for SQL Server)

<value> is a complete JDBC connection string. If this value is specified then it must appear last in the comma-separated list of <con-param> options.

Do not specify parameters instance, host and port because they shall be expected to be contained within the JDBC string value. If you need to specify the user and password (when using SQL Server authentication) then either specify these as separate parameters (above) or within the JDBC string value (but not in both places). If the project file already defines user and password values then those values shall be assumed unless you override them with the user and password parameters.

Note that in the examples below the new database name is specified twice. Once for the newname param (used by DataVeil for internal name substitution) and once within the actual JDBC string (databaseName=) which is passed to the JDBC connection manager.

Example:

This example shows a JDBC connection string when using Windows authentication. Requires auth=windows and the JDBC string requires integratedsecurity=true.

--con="{newname=stagedb1,auth=windows,threads=8,jdbc=jdbc:sqlserver://127.0.0.1:1433;databaseName=stagedb1;integratedsecurity=true;}"
 

Example:

This example shows a JDBC connection string when using SQL Server authentication. Requires auth=sqlserver.

--con="{prevname=kona,newname=k 2,threads=8,auth=sqlserver,user=sa,password=MkQ81zp5,jdbc=jdbc:sqlserver://127.0.0.1\\myinstance:1433;databaseName=k 2;}"

Note that the JDBC syntax requires a backslash to precede the optional instance name ('myinstance'); however, you must specify a double backslash because one backslash will be consumed as an escape character.

For further information on the syntax of the JDBC string for SQL Server please refer to https://docs.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url . If this address ceases to exist then try searching the internet for 'sql server jdbc connection string'.
  

native_prefix=<value>

<value> shall override the project file's Function call prefix setting for the connection.

Example: --con="{native_prefix=misc.dbo}"

password=<value>

For Oracle: <value> is the password corresponding to the current username. If the user parameter is not specified then the username defined in the project connection shall be assumed.

For SQL Server: <value> is the password corresponding to the current username. If the user parameter is not specified then the username defined in the project connection shall be assumed. This value shall only be used if auth=sqlserver is specified.

Example: --con="{prevname=db1,newname=stagedb1,password=m3VanQ562}"
  

port=<value>

<value> is an integer between 1 and 65535 inclusive.

Example: --con="{prevname=db1,newname=stagedb1,host=srv1.test.lan,port=1522}"

servicename=<value>

Applicable for Oracle connections only.

If servicename is specified then the sid and tnsname parameters must not be specified.

Example: --con="{servicename=svc01}"


 sid=<value>

Applicable for Oracle connections only.

If sid is specified then the servicename and tnsname parameters must not be specified.

Example: --con="{newname=myotherschema,sid=ora02}"
  

threads=<value>

<value> is the maximum number of masking threads to be used. Each thread corresponds to a JDBC connection to the DBMS from DataVeil. The valid range is from 1 to 10000 inclusive.

Note: For very short operations, such as those whose operations will complete within a few seconds, DataVeil may momentarily exceed this limit.

The recommended value is generally between two to three times the number of processors that are available to DataVeil that will not be performing any other significant processing during the masking run. For instance, if you have 6 processors that will not be busy with other tasks during a masking run then the threads <value> would typically be between 12 and 18. The benefit of having a higher number is faster execution time; however, if the value is set too high then you may overload your system resources and execution will become less efficient and take longer.

Example: --con="{prevname=db1,newname=stagedb1,threads=12}"

tnsname=<value>

Applicable for Oracle connections only.

If tnsname is specified then the servicename and sid parameters must not be specified.

Example: --con="{prevname=db1,newname=stagedb1,tnsname=tns03}"

user=<value>

For Oracle: <value> is an Oracle username.

For SQL Server: <value> is an SQL Server username. This value shall only be used if auth=sqlserver is specified.

Example: --con="{prevname=db1,newname=stagedb1,user=myuserid}"

 

 

--console {suppress|new}
  

By default, log messages shall be echoed to the console.

suppress prevents DataVeil log messages from being shown on the console.

new opens a new command window where DataVeil log messages shall be shown.

NOTE: Unlike other params, there must be a space (not an "=") between "--console" and its value ("suppress" or "new")

 

--createdirs={true|false}
  

Specifies whether directories should be automatically created if they do not exist.

This applies to any directories specified in --log, --logxml and --reportdir options.

If 'false' and a directory does not exist then DataVeil shall terminate without executing the project.

The default is true.

 

--determinism={det|non_det}
  

Overrides the project file's default determinism setting.

det specifies that the project default determinism shall be deterministic for this masking execution.

non_det specifies that the project default determinism shall be non-deterministic for this masking execution.
  

--det_ignorecase={true|false}
  

Overrides the project file's default deterministic 'Ignore case' setting.
 

--det_seed={auto|<default-seed-value>}
  

Overrides the project file's default deterministic seed setting.

auto specifies that the project default deterministic seed shall be automatically generated to a random value for this masking execution.

Any other value specifies an explicit seed value that shall be used as the default deterministic seed for this masking execution.
 

--diagnostics={true|false}
  

Specifies whether diagnostic messages should be written to logger locations.

Diagnostic messages are written to the console and any files specified in --log and --logxml options. Diagnostic messages shall not appear in reports.

The default is to use the preference setting from the GUI which can be found under menu Tools->Options:Messages. The default preference is false.

These diagnostic messages are intended for DataVeil Technologies' technical support staff. Therefore, they may not be meaningful to regular users. If you believe there may be a problem with the DataVeil software then please enable this diagnostics option and provide the resulting logger output along with a detailed description of the issue to DataVeil Technologies Support. If you are using the GUI then you can just right-click on the Output logger window and save the messages to a file.

 

--key=<project-key-value>
  

If the masking project requires a project key then --key must be specified.

If this option is specified and the project does not require a key then this option shall be ignored.

 

--license=<license-file-name>
  

If specified then DataVeil shall load the specified DataVeil license file.

If the file is not found or not valid then DataVeil shall terminate.

If this --license option is omitted then DataVeil shall attempt to load the license from the last known location as set by the DataVeil GUI for the current user.

 

--log=<file-name>[!]
  

If specified then DataVeil shall write log messages to a simple text file. The default directory is relative to the unzipped parent directory "dataveil".

If ! immediately follows <file-name> then the file shall be overwritten if it already exists, otherwise the log messages shall be appended. There must be no space before the ! char. 

 

--logioprogress={true|false}
  

Specifies whether messages that indicate IO progress within masking and table writer tasks should be written to the logger. Such messages shall generally indicate the task and project percentage completion status.

An example of such a message is as follows:

INFO ...progress of writing table [dvtest].[dbo].[t_4M]. Batch of 500,000 rows took 0,11 (mins,secs). Table write is 75% complete. Project is 92% complete.

The default is to use the preference setting from the GUI which can be found under menu Tools->Options:Messages. The default preference is false.

 

--logxml=<file-name>[!]
  

If specified then DataVeil shall write log messages in XML format to a text file. The default directory is relative to the unzipped parent directory "dataveil".

If ! immediately follows <file-name> then the file shall be overwritten if it already exists, otherwise the log messages shall be appended. There must be no space before the ! char.

 

--refreshschema={true|false}
  

Specifies whether to perform a refresh of the data location schema(s) in this masking project prior to attempting to mask the data. This will avoid the DataVeil 'database schema
 has changed' type of errors for changes that are relatively minor such as key or index definition changes that do not affect the masking definitions.

If a schema change involves the addition of dependencies on a masked column then DataVeil shall automatically mask the new dependants with no warnings. If the
 schema change involves the removal of dependencies then DataVeil shall correctly not mask the formerly dependent columns but they shall remain marked as sensitive, thus warnings shall be
 logged stating that these sensitive columns have not been masked. DataVeil shall report an error if it finds that a masked column is missing after the refresh.

The source project file shall not be modified to reflect any schema changes detected. If the option to save the project file together with masking reports is enabled then the exact copy of the original file shall be saved without any such changes. The actual report shall show precisely what was actually masked.

If the refresh operation fails then the masking run is canceled and no reports shall be generated. The log file, if specified in the run command, shall include details of the reasons
 why the refresh failed.

The default is true.

 

--reportdir=<directory-path>
  

If specified then DataVeil shall use the directory at the specified path as the parent directory for where each masking run child report directory shall be created.

If the specified directory path does not exist then --createdirs shall be examined. If true then the path shall be created. If false then the masking run shall be terminated.

If this --reportdir option is omitted then DataVeil shall use the last known directory path as set by the DataVeil GUI for the current user.

 

--reportkeep={<keep-count>|all}
  

If specified then DataVeil shall auto delete old masking reports except for the most recent <keep-count>. This refers to the other masking reports in the same directory as where the current masking run report is being written. 

If all is specified then no reports shall be deleted.

If this --reportkeep option is omitted then DataVeil shall use the last known setting as set by the DataVeil GUI for the current user under tab Execution->Reports->(setting cog icon).