Supplier deduplication is the sort of task that sits on every procurement improvement list and never reaches the top of it. The AP team already knows the duplicates exist, because a payment run pulls up two records for what is obviously the same vendor. The controller knows it matters, because it turns up in audit findings and distorts spend reporting. And still the cleanup gets deferred, quarter after quarter, because it looks like a large, boring data project with no obvious owner.
The deferral is expensive in a specific way. One plant buys from Bosch Rexroth. Another buys from Rexroth AG. A third buys the same components through a distributor. Your systems show three suppliers and three moderate relationships. The market shows one supplier and one relationship you have been negotiating from a weaker position than you needed to.
This piece covers how duplicate vendor records accumulate, which matching keys are reliable and which are traps, a sequence for running the cleanup safely, and how to keep the vendor master from drifting back within a year.
How Duplicate Vendor Records Accumulate
Nobody makes an obvious mistake. That is the frustrating part. Over time the same vendor enters the system under slightly different names, with slightly different addresses, and sometimes with different tax IDs or bank details depending on which subsidiary or division submitted the paperwork.
The common origins repeat across industries:
- A new ERP or a regional rollout creates a second record because the first was not migrated cleanly
- An acquisition brings its own vendor master, complete with overlapping suppliers
- A supplier restructures, and the legal entity you contracted with no longer exists, but the old record stays active
- A seasonal or promotional vendor is deactivated and then re-onboarded next year as a new record
- Someone needed a payment released on a Friday and set up a fresh record rather than chasing a change request
Manufacturing has its own variant. Raw material suppliers often run divisions that invoice independently: chemicals, plastics, packaging. Whether those should consolidate depends on whether you manage the relationship at parent or division level, which is a commercial decision rather than a data one. Deduplication work tends to surface these structural questions that the vendor master has been quietly obscuring.
The Matching Keys, Ranked by How Much You Should Trust Them
Not all fields deserve equal weight, and treating them as equal is how deduplication projects create damage.
Tax identification number. The strongest single key in most cases. A single legal entity should hold one tax ID, so multiple records sharing one is a clear duplication signal.
But the reverse does not hold, and this catches people out. A tax ID mismatch does not rule out a duplicate, because the same vendor may have submitted different tax IDs for different entities within its corporate structure. Treat a match as strong evidence and a mismatch as no evidence.
Bank account details. The same account number across two or more vendor records is a strong duplicate indicator. It is also, occasionally, a fraud indicator, which is a separate reason to review those matches carefully rather than merging them on sight.
Vendor name. Needs fuzzy matching, never exact string comparison. Acme Corp and ACME Corporation are the same entity, and a standard database query treats them as unrelated. Algorithms like Levenshtein, Jaro-Winkler and Soundex measure similarity rather than demanding character-for-character identity, which is why exact matching alone misses the majority of true duplicates.
Address. Weak on its own. Vendor addresses reflect whatever the supplier wrote on their tax paperwork or whatever an AP clerk typed at onboarding. Useful for confirming a candidate, unreliable for generating one.
Payment terms, currency, contact details. Supporting evidence only. Inconsistencies across these fields on otherwise similar records often reveal data entry errors made during onboarding, which is useful context when an analyst is deciding a borderline case.
A Sequence That Does Not Break Anything
The order matters more than the tooling. Running fuzzy matching first, on unstandardised data, produces a review queue nobody will finish.
Step one: extract everything, including the inactive records. Pull all active and inactive vendors from the ERP with vendor ID, name, address, tax ID, bank account, payment terms and status. Inactive records matter because they are frequently one half of a duplicate pair.
Step two: normalise before you match. Standardise legal suffixes, casing, punctuation and address formats. This is unglamorous and it does most of the work, because it converts a subjective matching problem into a partly deterministic one.
Step three: run exact matching first. Exact match on tax ID, VAT number, bank details, remittance email, or combinations of fields that should be unique. This is the safest and fastest sweep, producing very few false positives, and it typically strips out a substantial portion of the file. What remains is a much smaller set for the subjective work.
Step four: apply fuzzy matching with calibrated thresholds. Digital Ind. versus Digital Industries is the classic case. Threshold tuning is the whole game here. Practitioners commonly land around a similarity score of 85 and above for vendor names, paired with manual review rather than automatic merging.
Step five: have a human confirm before any merge. Automated matching surfaces candidates. It does not make the decision. A structured review reduces false positives and protects against merging legitimately separate entities, which is the failure mode that actually hurts, because an incorrect merge is far harder to unwind than a missed duplicate.
Step six: assign canonical IDs and keep the lineage. Every merged record should retain a trace back to its source records. When a category manager challenges a number six months later, you need to show the working.
Why False Positives Are the Real Risk
Most write-ups on this topic optimise for finding more duplicates. In practice the expensive error runs the other way.
A missed duplicate leaves you where you already were: understated spend with one supplier, a negotiating position you did not fully exploit. Annoying, recoverable in the next cleansing cycle.
An incorrect merge is different. Two genuinely separate legal entities collapse into one record. Payments route to the wrong bank account. Contract terms get applied to a supplier that never agreed to them. Audit trails break. And because the merge looks like a data improvement, nobody goes looking for it until something downstream fails.
This is why the review step is not optional, and why the tax ID logic above matters so much. The same corporate group can legitimately hold multiple entities with different tax IDs, different banking and genuinely different commercial relationships. Those belong in a hierarchy, linked but distinct, not merged into a single record.
Deduplication and Hierarchy Are Different Jobs
Worth separating clearly, because conflating them causes most of the damage.
Deduplication resolves records that represent the same entity. Fourteen versions of one supplier become one supplier.
Hierarchy links records that represent different entities under the same corporate family. Three subsidiaries stay three records, related by a parent-child structure, so you can report either way.
Entity resolution is the discipline that maintains both over time: a persistent unified identity across datasets, including a history of which source records contributed to each master identity. That persistence is the difference between a cleanup and a capability.
Get the sequence wrong and you either merge things that should have been linked, or you build a hierarchy on top of a duplicate-riddled base and inherit every error.
Keeping It Clean After the Project Ends
A vendor master drifts back toward chaos unless three things change alongside the cleanup.
Close the onboarding gap. Most new duplicates enter through vendor creation. A mandatory duplicate check at onboarding, against tax ID and fuzzy name match, prevents more records than any periodic cleanup removes.
Give the vendor master an owner. Not a committee. One named team accountable for record quality, with the authority to reject a creation request. Shared ownership means no ownership, and this is where most remediation efforts quietly unwind.
Monitor a small set of numbers. Duplicate rate as a percentage of active records. New vendor creations per month, with a look at any spike. Records with missing tax ID or bank details. Records inactive for two years and still open. Four numbers on a monthly report is enough to catch drift early.
The Twenty-Supplier Version
If a full vendor master programme is not going to get funded this year, and it often is not, there is a compressed version worth running anyway.
Take your top twenty suppliers by spend. Resolve their duplicates and corporate families manually, using tax IDs and a bit of desk research on corporate structures. It is roughly a week of analyst time.
Top suppliers hide the largest concentration surprises, because that is where the spend is. You will typically find at least one relationship materially larger than your reporting shows, which is exactly the finding that funds the wider programme. It also gives you a defensible number to walk into a renewal with, which is worth more than a tidy database nobody uses.
What to Check This Week
Three questions that will tell you the state of your vendor master without a project.
How many active vendor records do you hold, against how many suppliers you actually transacted with in the last twelve months? A wide gap is dormant records, duplicates, or both.
How many records share a tax ID or a bank account? A query, not an initiative.
And when someone asks what you spend with your largest supplier’s full corporate family, does the answer come from a report or from three people comparing spreadsheets?
The last one usually settles the argument about whether this work is worth doing.