Duplicate Value Rate

The Duplicate Value Rate analysis measures the percentage of duplicate values within a field.

Use it to identify fields with high redundancy that may benefit from deduplication or normalization efforts.

Duplicate Value Rate vs. Uniqueness Detection

The Duplicate Value Rate analysis tells you what percentage of records share a value with another record, while the Uniqueness Detection analysis only tells you whether a field is fully unique or not. A field can fail Uniqueness Detection while still having a low Duplicate Value Rate, if only a handful of records share a value.

Configuration

You might want to disable this analysis type for fields where it's normal and expected to have duplicate values, such as City, Country, Account Owner, etc.

Indicate whether the check for duplicate values should be Case Sensitive. When Case Sensitive is enabled, values such as aa and Aa are treated as two separate, unique values.

Set Thresholds for what constitutes a good, warning, or critical Duplicate Value Rate. For most fields you want as few duplicate values as possible. For example, a good rate might be 10% or less, warning between 10 and 30%, and critical 30% or higher.

Detailed Job Results

The Duplicate Rate per Field bar chart shows the percentage of records with a duplicate value. Hover over a bar to see more details for that field.

Below, the results table displays more detailed analysis results.

Records with Value is the number of records where this field contains a value.

Different Values is the count of distinct values found. For example, in a set of six records, with values [A, A, B, C, C, C] , the number of different values (distinct values) is three: A, B, C.

Records with Duplicate Value is the number of records that share a value with at least one other record for this field.

Duplicate Value Rate % calculates the number of records with a duplicate value for this field as a percentage of all analyzed records with a value. It is is color coded green, amber, or red based on the configured thresholds.

Key Insights

• Duplicate Value Rate: shows whether many records share the same values for a field, or whether most values are distinct.
• Level of non-uniqueness: a field can be non-unique, but still have a low duplicate value rate (only a few records with duplicate values). Or it is widely duplicated (high Duplicate Value Rate) and is therefore clearly unsuitable as a key or matching field.
• Input behavior: a high Duplicate Value Rate can point to default or placeholder values being reused, integrations writing the same value for many records, or users reusing codes meant to be unique.

Recommended Actions

Scenario	Actions
Field should be unique, but has a Duplicate Value Rate above 0%	- Identify the duplicated values and the records involved, then decide which record should keep each value. - Correct typos or placeholder values, and merge true duplicates with Plauti Deduplicate. - Add or tighten validation rules on the field to prevent new duplicates.
Field not intended to be unique, and Duplicate Value Rate is high	- Avoid using the field as a key in matching, integrations, or reporting. - Document the field as a segment or attribute rather than an identifier.
Protecting reporting, routing and AI	- Make sure fields with a Duplicate Value Rate of more than 0% are not used to represent unique entities in reports, routing and AI.