Every Dynamics 365 CE partner has a way of dealing with duplicate records. Nobody starts a migration planning to ship duplicates, and yet almost every project ends up with some version of a cleanup step bolted on, usually improvised, usually under time pressure, usually different from what the last project did.
We've written before about why that keeps happening project after project, and what the workaround rebuild actually costs across a year of migrations. This piece is the practical follow-up: a straight comparison of the five ways CE partners actually handle duplicates today, ranked by what each one really costs you once you count the hours, the risk, and what happens after go-live.
None of these are wrong, exactly. Some are just far more expensive than they look on a statement of work, and the expense rarely shows up where you'd expect it. It's not usually the line item itself that hurts. It's the hours that don't get billed because nobody scoped them, the callback three weeks after go-live when a rep notices the same account twice, or the version of this exact work you'll be doing again on the next project because nothing from this one carried over.
This is the default when there's no time to build anything else: export the relevant entities to Excel, eyeball or formula-match for duplicates, manually decide which record survives, then reimport or manually merge. It requires no setup and no tooling budget, which is why it survives on smaller projects.
It's also the most expensive option on this list once you count consultant hours. Matching is judgment-based, which means it's slow and inconsistent between whoever's doing it. It rarely accounts for related records, so re-parenting activities, opportunities, and cases after a manual merge becomes its own cleanup project. And because it's manual, it doesn't scale past a few thousand records without becoming the single biggest line item in the migration.
Real cost: high consultant hours, high risk of broken relationships, zero repeatability between projects. If you've ever quoted a migration and watched the cleanup phase quietly eat the margin you built in, this is usually why.
Setting alternate keys, usually against an external ID or a composite of fields like email and company, before an import is a legitimate way to prevent duplicates from entering during that specific migration. It's fast to configure and it catches the obvious cases at the point of load.
The problem is scope. Alternate keys only protect the import they're configured for. They don't catch duplicates created afterward by manual entry, by an integration, or by Dual-Write syncing from Finance and Operations. And they don't help with the duplicates already sitting in the environment before the migration started. It's a good tool for one job, not a data quality strategy.
Real cost: low setup cost, but a one-time fix that expires the moment go-live happens and normal operations resume.
Native detection rules and the bulk detection job are built into Dynamics 365 CE, free, and genuinely useful for catching obvious duplicates created through manual entry in the app. For a lot of smaller, simpler environments, it's a reasonable first line of defense.
Where it costs you is in the gaps. The bulk detection job caps out at 5,000 records per run, so scanning a full dataset means chaining jobs manually. Records created through API inserts, imports, or integrations can bypass detection rules entirely depending on how they're written. And detection isn't resolution: every match native tooling finds still needs a person to review and merge it, one record at a time, in the UI.
Real cost: no license fee, but ongoing manual review, blind spots at every non-UI entry point, and detection that quietly stops scaling once the dataset grows. It's the option most likely to give a client false confidence, since "we have duplicate detection on" sounds like coverage even when large parts of the environment are outside its reach.
We've covered this one in detail elsewhere because it deserves its own autopsy, but the short version: a custom flow built to catch duplicates on creation or on a schedule is a reasonable engineering response to native detection's gaps. It can be built to be smarter than the out-of-the-box rules.
It's also the option most likely to quietly break. Flows built for one client's field structure don't transfer cleanly to the next project, so most firms end up rebuilding a version of the same flow every time rather than reusing one. Schema changes, new fields, and new entities all risk breaking the matching logic silently, and unless someone is actively watching flow run history, a broken dedupe flow can go unnoticed for months while duplicates pile back up.
Real cost: real engineering time to build well, then ongoing maintenance risk that usually isn't billed to anyone, until it breaks and becomes urgent.
Some migration toolchains, ETL platforms like KingswaySoft among them, include fuzzy matching as part of the load process. It's a step up from alternate keys since the matching logic can be more forgiving, and it's already part of a toolchain many partners are using anyway.
The limitation is timing and scope. This kind of matching typically only runs during the migration itself. It's not native to Dataverse, doesn't run automatically as new records get created afterward, and stops protecting the environment the moment the project wraps and the toolchain gets put away until the next migration. It solves the go-live problem and leaves the same post-go-live gap every other workaround on this list has.
Real cost: solid coverage during the migration window, no coverage after it, and another vendor relationship and license to track.
Look at the list again and the pattern is obvious: every option either costs real hours, stops working after go-live, or both. None of them are native to Dataverse, repeatable across every project without rebuilding, and still running after the migration ends, all three at once.
That's not a coincidence. It's what happens when duplicate handling gets solved project by project instead of once, as a standard part of how a firm delivers Dynamics 365 CE work. If you're evaluating what would actually close that gap, that's the criteria worth starting from: something that works the same way on every project, keeps working after go-live without anyone rebuilding it, and doesn't require a new tool relationship every time your migration toolchain changes.
We'll get into what to look for in more detail in a later piece. For now, it's worth at least running the numbers on what your firm's current approach, whichever one of these five it is, actually costs you across a year of projects rather than one. Most firms have never actually added it up. When they do, the number tends to be a lot higher than the line item on any single statement of work suggests, because the real cost isn't concentrated in one project. It's spread across every project, quietly, in hours nobody's tracking against a single budget.
Blog
Most Dynamics 365 CE partners solve duplicate detection the same way on every migration, then rebuild it from scratch next time. Here's why, and what a one-time platform fix looks like.
Blog
Duplicate detection workarounds on Dynamics 365 CE migrations often go unscoped, and the hours come straight out of a boutique practice's margin. Here's what a year of engagements actually costs.
Five approaches show up most often: manual export/dedupe/reimport in Excel, alternate keys set during migration, native Dynamics 365 CE duplicate detection, a custom Power Automate dedupe flow, and bolt-on matching inside a migration toolchain such as KingswaySoft.
No. Native detection rules and the bulk detection job are built to catch duplicates created through manual entry in the app. The bulk job also caps at 5,000 records per run, and records created through API inserts, imports, or integrations can bypass detection rules depending on how they're written.
Flows built for one client's field structure rarely transfer cleanly to the next project, so most firms rebuild a version of the same flow every time rather than reusing one. Schema changes and new fields or entities can also break the matching logic silently, sometimes for months before anyone notices.
No. Alternate keys only protect the specific migration they're configured for. They don't catch duplicates created afterward through manual entry, integrations, or Dual-Write syncs, and they don't address duplicates already in the environment before the migration started.
Configured once per environment, covering every entry point. See how other CE practices restructured this.