Understanding Search Results

Discovery Search helps identify columns that may contain sensitive data.

Search results are based on the configured query, selected patterns, search options, sampled data, and discovery heuristics.

They should be treated as possible matches that require review, not as final classifications.

The built-in system patterns provide a useful general-purpose starting point. They are intended to help you quickly find common types of sensitive data, such as names, addresses, telephone numbers, email addresses, account numbers, identification numbers, and similar values.

However, system patterns are not a complete solution for every database. They do not know your application, your business rules, your internal terminology, or the intended meaning of each column. Application databases often contain specialized column names, internal codes, regional formats, abbreviations and business-specific meanings that cannot always be interpreted correctly by generic patterns.

For this reason, Discovery results should always be reviewed before columns are added to a masking project or before proposed masks are applied.

 

Why Results May Differ from Expectations

Some values can have more than one meaning. A value that is sensitive in one database may be not sensitive in another.

For example, a value may be:

* A person name in one database;

* A product name, color, category, or code in another;

* A valid identifier in one country or system, but ordinary numeric data elsewhere;

* Sensitive data that appears only rarely in the sampled rows.
 

Column names can also be ambiguous. Names such as "name", "code", "type", "value", "reference", or "description" may contain very different kinds of data depending on the application.

Discovery Search can only use the information available at the time of the search. This may include the column name, sampled values, configured patterns, confidence settings and false-positive filtering.

As a result, some columns may be classified differently than expected, assigned a lower confidence score, filtered out, or omitted from the results.

 

Use Custom Patterns for Application-Specific Data

If your database contains specialized categories of sensitive data or search results using the built-in system patterns are not satisfactory, you should create Custom Patterns.

Custom Patterns are intended for cases where the built-in system patterns are too broad or too generic for your application. They allow you to define patterns that more closely match your actual column names and data formats.

Custom Patterns are especially useful for data such as:

* internal customer, member, policy, claim, account, patient, student, or employee identifiers;

* proprietary reference numbers;

* application-specific codes;

* regional identifiers not covered by the system patterns;

* columns that follow known internal naming conventions.

For best results, a Custom Pattern should reflect how the data actually appears in your database. Where possible, define both a column name pattern and a data value pattern. Matching both the column name and the stored data usually gives more accurate results than matching on either one alone.

 

Search Options Can Affect Results

Search Options can significantly affect what appears in the Discovery results.

For example, the following options may change whether a result is shown, hidden, or classified in a particular way:

* Filter False Positives

* Minimum Confidence %

* Sample size

* Search Project Columns

* Search DBMS Columns

* Search Excluded Columns

* Ignore Columns marked Sensitive

* Ignore Columns marked Not Sensitive

If an expected result is missing or appears to be classified incorrectly, review the Search Options before assuming that the pattern itself is wrong.

 

Filter False Positives

The Filter False Positives option deserves special mention because it can significantly influence the Discovery result.

This option is intended to reduce clutter by suppressing results that are more likely to be false positives.

This is useful when using broad system patterns. However, it can also hide or affect results in some specialized cases, especially when there are only a small number of rows available for analysis.

For example, suppose a column is used to store person surnames, and the sampled values include names such as:

* Rose

* Violet

* Ruby

* Amber

These may be valid person surnames in your application. However, they are also common words in other contexts, such as flowers, colors, gemstones, or descriptive labels.

If the column contains only a few rows, Discovery may not have enough data to distinguish between these possible meanings. If Filter False Positives is enabled then the result may be hidden or classified differently than expected.

When testing Custom Patterns, especially against small tables, consider disabling Filter False Positives. This prevents built-in false-positive filtering from hiding or reclassifying results that your Custom Patterns were specifically written to find.

 

Sample Size and Representative Data

Discovery Search normally examines a sample of rows rather than every row in every table. This helps keep searches practical and efficient.

However, results depend partly on whether the sampled rows are representative of the column as a whole.

Unexpected results may occur if:

* sensitive values do not appear in the sampled rows;

* the sample contains only unusual or ambiguous values;

* test data is sampled instead of realistic data;

* a column contains mixed formats, but only one format appears in the sample.

If the results appear incomplete or inconsistent, consider increasing the sample size or reviewing the column values directly.

 

Reviewing the Result Summary

After running a Discovery Search, review the Result Summary carefully. Pay particular attention to the classification, confidence score, matching pattern, matching values and rejected values.

You can also click on a column name in a Result window and further review data samples for that column in the Data Browser.

These details can help you decide whether a result is a true positive, a false positive, or a partial match that requires a different pattern or masking strategy.

Discovery Search is a useful convenience feature, but the final decision should be based on user review and knowledge of the application database.