Deduplicating a CRM at Scale When the Platform Gives You No Bulk Merge

Company, people and exact figures are withheld; proportions are real.

The situation

A B2B research firm inherited a CRM company object in the low six figures. Over a third of those records were surplus — the same company entered two, five, sometimes seventy times. The instinct was that the sync from the firm’s system of record was leaking duplicates in. It wasn’t. Records carrying a system-of-record key were 99.7% clean. Nearly nine in ten surplus records had been created by the CRM itself: contact imports and workflows that had no domain to match on, so every contact with a company name minted a fresh company. Two events accounted for most of it — a list-upload workflow that created tens of thousands of companies in one day, and a single contact import that created twenty thousand in two minutes.

The cost was concrete: a large share of lead handoffs were failing because contacts were attached to the wrong one of several identical company shells.

Why nobody had fixed it

Not for lack of a remedy. For lack of a route to it.

• The system of record could not be written to by the ops team, and dedupe tooling there was off the table on budget.
• The CRM’s native duplicate manager is a human review queue. Its bulk merge works only on a manual selection, fifty rows a page, selection lost on every search, criterion reset every time. Custom detection rules cap at about a thousand groups with no search and no filters, and do not expose record IDs.
• Import cannot merge. Workflows cannot merge. The vendor’s AI assistant had no merge capability at all — not a permissions gap, a missing tool.
• Nobody owned the problem end to end.

The one opening

A workflow custom-code action can call the merge endpoint. It is the only place in the platform where a merge happens without a click. Everything else in the design exists to feed that opening safely.

Decide offline, execute inside the platform. A full export was evaluated against the entire company object — the step the UI cannot do — to produce candidate pairs. A new text property on the company, merge into record ID, became the bridge: an import writes the survivor’s ID onto the record that will disappear, and nothing else in the portal uses the property. A workflow enrols on that property being known, runs the guarded code, and routes every non-merge to a static review list.

The credential never leaves the platform. The API key lives as a workflow secret and the code reads it from the environment. Merge calls originate from the vendor’s own servers under an app the portal owner created. There is no external system in the call path.

The higher-stage record is always the survivor. A merge adopts the furthest-down-funnel lifecycle stage, and that is not configurable. If the survivor were the lower-stage record it would be promoted and stamped with a fresh date entered — corrupting funnel timing. Making the higher-stage record the survivor means its stage never changes and no new stamp is written. This was verified empirically, not just by construction.

Twelve guards, on live data. Both records are re-fetched from the API at run time — not read from the CSV — so anything that changed since analysis is caught. Keyed records, deals, owners, hierarchy, country, domain, free-mail domains, unrecognised stages, unapproved stages, and orientation. The default outcome of an unclear case is inaction with a reason written out.

The gates

Nothing ran at scale until it had run small.

• Gate 0 — three rows that could not merge: a non-numeric target, a self-reference, and a real pair inverted on purpose to fire the orientation guard. Proved the plumbing end to end at zero risk.
• Gate 1 — one real merge, checked field by field.
• Gate 2 — ten pairs spread across stage combinations and countries.
• Batch A — a stratified 250, mirroring the full mix.

The workflow was off between every gate. Every batch was authorised explicitly. Every irreversible action had a captured before-state.

What the platform does that the documentation does not say

Five behaviours, each established from evidence rather than from the manual:

1. Property precedence: the surviving record’s values win. The merge dialog says “in most cases the most recent value for each property will be used.” Across five records with a deliberately chosen discriminator field, the survivor’s value was retained every time. The knowledge base is right; the dialog is misleading.
2. The survivor inherits the label. The merge copies the loser’s merge into record ID onto the survivor, leaving it pointing at itself. Harmless once, a review-list flood at scale. The fix has to live inside the code — see 3.
3. No downstream workflow action runs for a merged record. The enrolled record ceases to exist the instant the merge lands, and the run stops there. Anything that must happen on success — clearing the label, writing a log — has to happen inside the code before the callback.
4. Date entered current stage follows its own rule. It takes the latest entry into the current stage across the merged history; but if that history contains a later entry into a different stage, the field is restamped to the merge time. This affected 29% of the candidate pool with stamps a median of eight months old. It was quantified and split into a separate, explicitly-accepted batch rather than discovered in a report later. The first hypothesis for the mechanism was wrong and was corrected against a record that refuted it.
5. Record IDs are preserved through the API merge endpoint once the relevant beta is enabled, and a merged survivor inherits the secondary’s enrolment history — so re-enrolment-off is honoured across a merge.

What went wrong, and what it taught

Two screening gaps in the pipeline — a domain-level clearance applied to one candidate set but not the other, and the restamp exposure — were caught before they became roughly 1,600 and 600 irreversible mistakes. Both were caught the same way: by building the before-state view for the next gate and noticing the numbers were wrong.

A partial batch that reported success. Of the 250 in Batch A, 46 merged and 204 landed in the review list with no error anywhere. Offline re-testing showed all 250 pairs were valid. The custom-code action simply had not executed for the 204: the action’s rate limit was off, so records that exceeded the platform’s internal throughput continued to the next step with no code result. The review list caught them — which is the point of having one — but nothing raised a flag. The tripwire now is one comparison after every batch: code executions must equal enrolments.

A retry the platform would have done for free, defeated by defensive code. The action’s catch block returned 429s and 5XXs as output. From the platform’s side that is a success, so its built-in three-day retry never fires. The fix is to let transient errors throw.

A rule that lived in the CSV instead of the code. The boundary between the approved stage jumps and the unapproved ones was real, documented and signed off — and enforced only by which rows were in the file. One extra guard closed it. The general lesson: every safety rule that matters belongs in the guard set, where a different file on a different day cannot bypass it.

Where it stands

Seventy merges executed, every one verified: no contact lost, no survivor’s stage changed, no per-stage stamp moved, no keyed record touched, no country crossed. A mechanism that refused 204 records rather than forcing them through. A candidate pool narrowed honestly from 3,700 to 1,461 — smaller, with every exclusion written down and held for its own decision. And a working proof of concept to take to leadership for the decisions that are theirs to make.

The larger point is upstream. The portal was generating duplicates faster than any cleanup could remove them. Cleaning without closing that hole is bailing a boat. The executor bought the credibility to insist on an import standard — a required domain column, no auto-creation of companies — before the next purchased list lands.

If you are doing this yourself

• Put the credential in a workflow secret. Never let it leave the platform.
• Make the higher-stage record the survivor. Always.
• Re-fetch both records at run time. Never trust the CSV.
• Enforce every safety rule in the code, not in the file.
• Turn on the action’s rate limit before the first batch above fifty.
• Let 429 and 5XX throw.
• After every batch, compare code executions to enrolments.
• Keep re-enrolment off, and know that it means held records need a deliberate retry.
• Expect the survivor to inherit the label; clear it inside the code.
• Quantify the date entered current stage exposure before you run; decide it on the record.
• Run three rows that cannot merge before you run one that can.

If you are inheriting a portal

This is what an inherited portal looks like from the inside, and why the native tools stop short of fixing it. The first step is never a rebuild; it is a read-only, evidence-based audit that tells you what is real, what created the mess, and what a fix will actually require. That is the Inherited Portal Audit — three weeks, credited in full against the fix. Or send a brief and I’ll tell you on the first call whether you need one.