How do I run multiple instances of DataVeil concurrently?

The default installation of DataVeil allows for only a single instance of DataVeil to be running at any given time in the same operating system user account. This is a combined total regardless of whether DataVeil is launched as a GUI or from the command-line.

The reason behind this is that DataVeil is built on the Netbeans Platform framework that uses a single user cache directory (“userdir”) for the application. Therefore, whenever another DataVeil instance is started, Netbeans attempts to use the same userdir but detects a lock from another instance of DataVeil and then refuses to run. This can produce an error message in a popup window and/or logged to the command-line console, as shown below. It can also cause the DataVeil instance that was already running normally to terminate unexpectedly.

mult_dv_error2

 

For most users, a single instance limit is adequate and this is all that a single-user license allows.

For users who have a multi-user license and would like to run multiple instances of DataVeil from the same operating system user account then the remainder of this article shall explain how this can be done.

As mentioned, the issue is the Netbeans userdir to which a DataVeil instance requires exclusive access. The location of this userdir can be set at DataVeil startup using the Netbeans parameter --userdir. See http://wiki.netbeans.org/DevFaqUserDir

You can see the location of the userdir of your current DataVeil installation in the About dialog of the DataVeil GUI.

mult_dv_about

 

Running multiple command-line instances of DataVeil

By default, a command-line instance of DataVeil shall attempt to use the same Netbeans userdir as the DataVeil GUI which means that both cannot be running at the same time.

Let’s change this so that the command-line instance uses its own separate Netbeans userdir.

This is done by including the Netbeans --userdir parameter in the DataVeil start command.

As an example, let’s consider the sample command line files in the DataVeil software /batch directory. In the original Window’s batch file, the start command appears as:

START "MyWindowTitle" /WAIT "..bindataveil64.exe" ^
--nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms256m -J-Xmx768m ^
--project="S:/Demo v1/SqlServer/kona_V4.dvp" ^
--key="123456" ^
--refreshschema=true ^
--compilewarning=continue ^
--createdirs=true ^
--log="mybatchlogs/mylog.txt!" ^
--logxml="mybatchlogs/mylog.xml!" ^
--logioprogress=true

Notice that there is no --userdir parameter which means that a default location has been chosen where all instances shall try to use.

Let’s change that by explicitly specifying a --userdir directory to use C:\dataveil\userdirs\user02

On Windows, the full directory path shall automatically be created if it doesn’t already exist. On *nix systems the directory path up to but not including the actual user dir (C:\dataveil\userdirs\) should already exist.

You may notice in the start command that I’ve used forward-slashes. Windows recognizes these and it avoids issues where in some cases double back-slashes would be needed if enclosed within quotes, which are needed if a space is embedded in the path… so I personally find it easier to use forward-slashes instead and it also makes it easier if I want to start with a copy of the command on a non-Windows system such as Linux.

Also notice that I have added the DataVeil --license parameter. This is because each separate userdir will persist its own set of environment variables and settings such as the license location. A default license location is stored in a userdir only from having run the GUI using the userdir. Therefore, a new userdir location would have an undefined license location and so it must be specified as a command-line parameter.

START "MyWindowTitle" /WAIT "..\bin\dataveil64.exe" ^
--nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms256m -J-Xmx768m ^

--userdir "C:/dataveil/userdirs/user02" ^
--license="S:/dataveil/licenses/license.dvl" ^

--project="S:/Demo v1/SqlServer/kona_V4.dvp" ^
--key="123456" ^
--refreshschema=true ^
--compilewarning=continue ^
--createdirs=true ^
--log="mybatchlogs/mylog.txt!" ^
--logxml="mybatchlogs/mylog.xml!" ^
--logioprogress=true

Note there is a space (no ‘=’) after --userdir whereas there is an ‘=’ after --license.

The line continuation character ^ requires at least one space before it and absolutely no characters after it, not even whitespace, otherwise Windows will throw an error, resulting in a cryptic error message. For example, by placing a single space character after the ^ in ‘--userdir "C:/dataveil/userdirs/user02" ^ ‘ above, the following error message would be produced:

'--license' is not recognized as an internal or external command, operable program or batch file. Unexpected exit code = 9009, probably a problem with the command or the environment.’

For *nix systems, the procedure is the same except that the --userdir parameter is set in the relevant *nix start command. For example:

../bin/dataveil --nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms256m -J-Xmx768m --userdir "/export/home/dataveil/userdirs/user02" --license="/export/home/dataveil/licenses/license.dvl" --project="/export/home/test/Documents/dataveil/Demo v1/Oracle/kona_sol1.dvp" --key="123456" --refreshschema=true --compilewarning=continue --createdirs=true --log="mybatchlogs/mylog.txt!" --logxml="mybatchlogs/mylog.xml!" --logioprogress=true

Using static batch files

A simple approach could be to create specific batch files where each explicitly references a different userdir.

For example, if you have a license for 3 concurrent users and wanted to run 3 DataVeil masking projects concurrently from the command line, you could create 3 copies of the batch file where each file would specify a distinct userdir.

i.e.
start_dataveil_01.bat could use --userdir "C:/dataveil/userdirs/user01
start_dataveil_02.bat could use --userdir "C:/dataveil/userdirs/user02
start_dataveil_03.bat could use --userdir "C:/dataveil/userdirs/user03

All 3 of these batch files can be started concurrently thereby executing 3 DataVeil masking projects concurrently.

Using dynamic commands

An alternative to using specific batch files where each runs using a fixed name userdir is to use an environment variable.

Therefore, in Windows, you could create an environment variable called DATAVEIL_USERDIR.

The START command could then be modified to use the environment variable. Immediately prior to running the batch file the environment variable can be set to an unused location, for example:

set DATAVEIL_USERDIR="C:/dataveil/userdirs/dynamic_148"

Now the --userdir parameter in the START command would reference the environment variable:

START "MyWindowTitle" /WAIT "..\bin\dataveil64.exe" ^
--nosplash --nogui -J-Dnetbeans.logger.console=true -J-Dorg.level=WARNING -J-Xms256m -J-Xmx768m ^
--userdir %DATAVEIL_USERDIR% ^
--license="C:/dataveil/licenses/license.dvl" ^
--project="S:/Demo v1/SqlServer/kona_V4.dvp" ^
--key="123456" ^
--refreshschema=true ^
--compilewarning=continue ^
--createdirs=true ^
--log="mybatchlogs/mylog.txt!" ^
--logxml="mybatchlogs/mylog.xml!" ^
--logioprogress=true

Therefore, you could have a system that either manages the allocated DATAVEIL_USERDIR folders, such as a round-robin large enough not to reuse a userdir that may still be in use, or you could simply increment the allocated userdir name (‘C:/dataveil/userdirs/dynamic_148’, ‘C:/dataveil/userdirs/dynamic_149’, ‘C:/dataveil/userdirs/dynamic_150’, and so on) and periodically run a cleanup job to delete the userdir folders that are no longer in use.

Clearly, using an environment variable provides most flexibility because the same command is used each time.

Resources

Memory: Each DataVeil instance shall allocate its own memory, as shown by the ‘-J-Xms256m -J-Xmx768m’ parameters in the START command, so please ensure that your system has adequate memory resources.

Compute: A DataVeil instance consumes relatively little compute resources during masking execution. This is because DataVeil uses the DBMS servers to perform all masking computations whereas a DataVeil process acts as a controller and remains mostly idle.

 

Summary

To run multiple concurrent DataVeil processes in the same operating system user account, you just need to allocate a separate Netbeans userdir for each instance.

You can do this by specifying the --userdir parameter in the DataVeil START command with an explicit path or by using an environment variable for more flexibility.

Sufficient memory must be available for all concurrent DataVeil processes. DataVeil moves the processing onto the DBMS during masking and therefore the actual DataVeil processes will consume minimal compute resources.