Skip to content

D365 CE partners keep rebuilding the same duplicate detection fix. Nobody seems to mind. They should.

  • Dynamics 365
  • Deduplication
  • Consulting
D365 CE Partners Rebuilding Same Fix. Photo by Mario Gogh on Unsplash

You close out a migration. The duplicate workstream took longer than scoped - it always does - but you got there. Post go-live, the data looks clean. The client signs off.

Six months later, a new project starts. Different client, similar stack: legacy CRM, a couple of integrations, Dual-Write. The project plan goes up on the board, and by week three someone flags duplicates in the migrated data.

You know exactly what to do. You've done it before. You write the plugin, set the header, run the cleanup job. It works. The client signs off.

Then it happens again on the next project.

At some point the question stops being "how do we fix this" and starts being "why are we fixing this the same way every time?"

The workaround that travels with every project

Most Dynamics 365 CE partners land on a version of the same solution:

  • A plugin registered on the Create message for the entities that matter most: Account, Contact, Lead.
  • A header flag set on the primary integration connector.
  • A post-migration bulk detection job to catch what got through.
  • Sometimes a Power Automate flow for ongoing deduplication between syncs.

It works well enough. Someone who understood the problem built it. It gets copied from project to project, adjusted for the new client's entity model, and deployed. It does the job.

The problem isn't that it doesn't work. The problem is what it costs to maintain at scale, and what happens the moment a project deviates from the template it was built for.

What "rebuilding it" actually means

Rebuilding the workaround on a new project isn't copy-paste. Each implementation involves:

  • Entity scope decisions. The plugin covered Account, Contact, and Lead on the last project. This client has three custom entities that feed into the same deduplication problem. Someone has to decide whether to extend the plugin, write new ones, or leave those entities uncovered and document the gap.
  • Integration-specific configuration. The header flag worked on KingswaySoft. This project uses a different ETL tool or a custom Web API client the integration team built in-house. The configuration doesn't transfer. Someone has to find the equivalent setting, or discover there isn't one.
  • Environment deployment. The solution has to be packaged, deployed into the client's environment, tested, and handed over. That's billable time on every project, for a problem that hasn't changed since the last one.
  • Post-migration cleanup. The bulk detection job still caps at 5,000 records. The dataset is larger than that. The batching has to be managed by hand, with someone tracking what's been processed while new records keep arriving through active integrations.

All of this takes time, and it gets scoped, estimated, and delivered again on the next project, and the one after that.

The cost that doesn't show up in the project budget

The direct cost is visible: hours spent on the duplicate workstream that show up on timesheets and get billed or absorbed depending on how the project is scoped.

The indirect cost is harder to see and larger over time.

Every hour spent rebuilding a known solution is an hour not spent on the work that differentiates your practice: configuration, data modeling, business process design - the work clients pay for and remember. Duplicate detection is infrastructure. It should behave like infrastructure: configured once, running in the background, not a workstream on every engagement.

There's also a knowledge dependency. The person who built the original workaround knows how it works. When they move to a different project, go on leave, or leave the firm, the next person inherits something that was never documented as a product - because it wasn't supposed to be one. It was a workaround.

Why the workaround never becomes a product

The natural response to this pattern is to productize the workaround: package it as an internal tool, maintain it properly, deploy it consistently. Some larger practices have done exactly this.

The problem is that the workaround sits on top of native Dataverse behaviors that weren't built for this use case. The plugin needs maintenance every time Microsoft updates the platform. The header flag depends on the calling system, so every new integration is a new dependency. The bulk detection job's cap isn't something you can engineer around - it's a platform limit.

You can make the workaround more reliable. You can't make it not a workaround.

What changes when you stop rebuilding

The alternative isn't a more sophisticated workaround. It's duplicate prevention that runs inside Dataverse at the platform level, so it doesn't depend on individual integration configuration, doesn't require per-entity plugin maintenance, and doesn't hit a ceiling on bulk jobs.

Plauti installs from Microsoft Marketplace and runs inside the client's Dataverse environment. It covers every entry point - UI, Web API, bulk imports, and integration syncs - without configuring each one separately. Standard and custom tables are covered out of the box, and there's no 5,000-record ceiling.

For partners, the practical shift is that the duplicate workstream stops being a project deliverable and starts being an implementation step: install, configure once for the client's data model, and move on. The configuration carries across engagements instead of being rebuilt from scratch each time.

That's the difference between a workaround and infrastructure.

What to do differently on the next project

Before the duplicate workstream gets scoped on the next migration, it's worth asking how much of what you're planning to build solves a new problem versus re-solving the same one.

If the entry points are familiar, the entity coverage is familiar, and the cleanup process is familiar, that's the signal the workaround has run its course. The fix at that point isn't a better workaround. It's moving duplicate detection out of the project plan and into the platform, once, so it doesn't come back on the next migration.

Hungry for more?

Frequently asked questions

Why do Dynamics 365 CE partners keep running into the same duplicate detection problems on every migration?

Native D365 CE duplicate detection was built for UI-based data entry, not for the mix of bulk imports, Web API calls, and integration syncs that go-live week involves. Partners typically solve this with a combination of plugins, header flags, and post-migration cleanup scripts, but those solutions are scoped to specific entities and connectors, so they have to be rebuilt and reconfigured on every new project rather than carried forward as reusable infrastructure.

What are the limitations of using a custom plugin for duplicate detection in Dynamics 365?

Plugins registered on the Create or Update message cover the entities they're written for, but don't automatically extend to custom tables or new entities added during the project. They also require packaging, deployment, and testing in each client environment, and need to be maintained as Microsoft updates the platform. For practices running multiple migrations a year, the cumulative maintenance overhead adds up quickly.

Is there a way to handle duplicate detection in Dynamics 365 without rebuilding the solution on every project?

Yes. A native Dataverse deduplication layer, installed once per client environment from Microsoft Marketplace, covers all entry points and entity types without requiring per-integration configuration. Rather than scoping duplicate detection as a project workstream on each engagement, partners configure it once during implementation and move on. The tool handles ongoing deduplication across UI, API, and integration entry points consistently.

What is the real cost of duplicate workarounds for a Dynamics 365 CE practice?

The direct cost is the hours spent building, deploying, and cleaning up on each project. The indirect cost is the knowledge dependency - a workaround built by one consultant that the next person inherits without documentation - plus the opportunity cost of time spent on infrastructure rather than differentiated delivery work. Practices running five or more migrations a year typically find that the cumulative cost of the workaround exceeds the cost of replacing it.

Ready to take control?

With a product tour you can walk through the product yourself without installing anything, or book a demo for a guided look.