Deduplicating 500,000+ Dynamics 365 Records at Scale

Five hundred thousand records spread across multiple business units isn't an edge case. It's the reality for mid- to large-sized enterprises running Dynamics 365 as their system of record. At that scale, the question isn't whether you have duplicates. It's how bad the problem already is, and what you can do about it without grinding your system to a halt.

Across organizations using Plauti to check their Dynamics 365 data, a significant share of incoming records get flagged as duplicates, in many cases exceeding 40%. At 500,000 records, that could mean over 200,000 you can't fully trust. If your organization spans multiple business units with separate data entry points, the duplication rate compounds across every integration, import, and manual entry.

Why can't Dynamics 365's native tools handle this volume?

Dynamics 365 ships with built-in duplicate detection. For small datasets and simple scenarios, it works well enough. At 500,000 records across business units, it falls short in ways that matter.

Native duplicate detection in D365 has a hard cap of 5,000 duplicates per bulk detection job. If your dataset exceeds that threshold, and at this scale it will, the job either stops or returns incomplete results. That's not a configuration issue. It's a platform limit.

There's also no fuzzy matching. Native rules support exact matching, first-N-character matching, and last-N-character matching. That means "Jon Smith" and "Jonathan Smith" won't be flagged as the same person, even if they share an email, a phone number, and an account. In real enterprise data, records rarely duplicate cleanly: people type names differently, companies get entered under multiple abbreviations, and phone numbers appear with or without country codes.

Once duplicates are found, there's no bulk merge either. Each record pair requires individual review and manual action. At 500,000 records, that's not a workflow. It's a full-time job for months.

What actually matters when deduplicating at this volume?

Four capabilities determine whether a deduplication tool is viable for a large Dynamics 365 deployment: fuzzy matching, safe bulk processing, cross-entity logic, and automation that keeps manual review manageable.

Fuzzy and phonetic matching

The tool needs to catch near-matches, not just exact ones. "Acme Corp" and "ACME Corporation" should resolve to the same entity. So should slight name variations, transposed digits in phone numbers, and alternate email formats. Plauti's matching library for Dynamics 365 covers fuzzy text matching, Company Name algorithms that absorb abbreviations and suffixes, Person Name matching that combines phonetic comparison with Levenshtein distance, and Phone Number normalization that treats "(415) 555-2761" and "+1 415 555 2761" as identical.

Bulk processing without performance impact

Running a dedupe job on 500,000 records shouldn't lock up the system for active users. Plauti's matching jobs run outside of Dynamics 365, either through Plauti Cloud (a dedicated, fenced-off Azure environment where each job gets its own isolated server) or on your own machine or server. Your D365 environment stays responsive while a job processes, and results come back only once it's finished.

Cross-entity logic

In D365, duplicates don't just live within a single entity. A Lead might duplicate a Contact. A Contact at one business unit might duplicate a Contact at another. Plauti's cross-entity deduplication for Dynamics 365 checks Leads against existing Contacts and Accounts, both in real-time prevention and in batch jobs.

Automation that reduces manual review

At 500,000 records, even reviewing 10% manually is 50,000 decisions. Plauti's auto-merge feature runs as a follow-up step on a completed matching job. You set a minimum matching percentage; records scoring above that threshold merge automatically, and records below it stay in the results for manual review. The queue shrinks to a fraction of the total.

How Plauti handles high-volume Dynamics 365 environments

Plauti for Dynamics 365 is built natively on the platform. No data leaves your environment through unsecured channels. When a job runs on Plauti Cloud, data is exported and imported via encrypted connections, and the job server is terminated once processing completes. Even Plauti can't access your data while a job runs.

Scenarios and matching methods

You configure Scenarios per entity: combinations of fields, matching methods (exact or fuzzy), field weights, and score thresholds. A Contact Scenario, for example, might combine Email (exact), Full Name (Person Name fuzzy), and Company (Company Name fuzzy), weighted so email carries the most influence. Two records score as duplicates only if the combined weighted score meets or exceeds your threshold, which keeps detection precise and explainable: admins can see exactly which fields drove a match.

Merge Rules

When duplicates are found, Merge Rules decide what happens. You configure, per entity, which record becomes the master: the oldest record, the most recently updated, a specific field value, or a specific business unit's record. You also define what happens to losing records, deactivate or delete. Related records (Activities, Opportunities, and so on) get reparented to the master, so no history is lost.

Auto-merge

After a matching job finishes, auto-merge can run as a follow-up step. Set a minimum matching percentage, choose where to run it, and the system merges every pair at or above that score. Lower-scoring pairs stay in the results for manual review. This is the only realistic way to close out a project involving hundreds of thousands of records.

Prevention after cleanup

Once historical duplicates are resolved, prevention stops new ones from forming. You configure, per entity, where duplicate checks run: on form save, as-you-type while a user fills in a form, on Quick Create, and on API or import inserts. For high-confidence API matches, incoming records can be merged automatically before they ever hit your database. Lower-confidence matches get routed into a review queue instead.

For multi-business-unit environments, Plauti handles cross-business-unit visibility so duplicates that exist across organizational boundaries don't slip through. Because everything runs inside Dynamics 365/Dataverse, there's no external ETL pipeline to maintain, no sync delay, and no separate application for your team to learn.

Structuring a deduplication project at 500,000 records

Even with the right tool, a project of this size benefits from a phased approach.

Start with a data audit

Before merging anything, understand the scope. Run a matching job, or several, filtered by entity and business unit, to quantify duplicates. This surfaces the highest-concentration areas, shapes your merge strategy, and gives you a before-and-after baseline to measure progress.

Define Scenarios and Merge Rules before you run bulk operations

Decide which matching methods apply to each entity, set field weights, and configure your score thresholds. Then define Merge Rules: which record wins, what happens to related data, and how losing records are handled. Document these decisions so they're applied consistently and can be reviewed later.

Run matching jobs in stages

Don't try to process all 500,000 records in one job. Filter by entity, business unit, or record creation date range. This keeps jobs manageable, makes errors easier to isolate, and lets you validate results before moving to the next segment. Plauti Cloud can run multiple jobs in parallel, which speeds this up considerably.

Use auto-merge for high-confidence pairs

Set a conservative minimum matching percentage for your first run, 90% or higher, say. Review a sample of the merged records to confirm accuracy, then gradually expand to lower thresholds as confidence grows.

Enable prevention after cleanup

Once existing duplicates are resolved, turn on prevention for new records. Cleaning 500,000 records and then letting duplicates re-enter freely wastes the effort. Configure prevention on Leads, Contacts, and Accounts at a minimum: on form save, Quick Create, and API insert.

Why does manual review stall so many deduplication projects?

This is where most deduplication projects actually stall. A matching job finds 80,000 potential duplicates. Someone has to review them. Nobody has time, and the project sits in a queue for months until someone declares the data "good enough" and moves on.

The fix is smarter triage, not more reviewers. A well-configured setup auto-resolves the easy cases, same email, same name, same phone, and only surfaces genuinely ambiguous records for human judgment. Setting a high auto-merge threshold handles the obvious cases without manual input, leaving a much smaller queue of records that actually need a human decision.

If you're managing 500,000 records across business units and need deduplication that actually finishes, native D365 tools won't get you there. They weren't built for this volume, this complexity, or this scale of ongoing prevention. Plauti was.