CSV Deduplicator

[ DEDUPLICATED CSV ]
email,name,phone,city [email protected],Juan Cruz,09175551234,Cebu [email protected],Ana Reyes,09281234567,Davao [email protected],Mark Santos,09171112222,Cebu
5 rows in · 3 kept · 2 removed
Rows in5
Kept3
Removed2
Conflicts1
Compare on
[ WHAT MATCHED ]
[email protected]rows 2, 4 · kept 2
[email protected]rows 3, 6 · kept 3conflict
1 group of rows matched on the key columns but differ somewhere else, so that group is not really a duplicate — they are two records about one thing, holding different values. Keeping the first one throws the other away without anybody reading it. They are listed below by row number; reconcile them by hand before you rely on this.
Rows were compared on email only. Everything outside those columns was ignored when deciding what counts as the same row — which is what makes a key useful and is also how a row with real data in it gets dropped.
Values were compared after trimming the ends, ignoring case. The original text is kept in the output — only the comparison was relaxed.
[ WHAT THIS IS ]

“Duplicate” is not a fact, it is a decision. Juan Cruz, juan cruz and Juan  Cruz  are three strings, one person, and possibly three customers, depending on where the file came from.

A tool that compares raw text finds none of them. One that normalises hard merges people who are genuinely different. Neither is right in general, so both are reachable here and the page says which one produced the number.

But the more useful thing is not the count, it is the conflict.

When two rows match on the key and differ outside it — one email, two phone numbers — that is not a duplicate. It is two records about one thing, and keeping the first is a coin toss dressed up as a rule. Every such group is listed by row number, so a person makes the choice.

Nothing is silently dropped. Every removed row is listed under the group it belonged to.

[ QUESTIONS ]

Is my spreadsheet uploaded anywhere?

No. There is no upload and no file picker — you paste the rows in and the work happens in your browser, in this site’s own calculation engine. Nothing is transmitted, logged or stored.

Why did it not find the duplicates I can see?

Almost always because of how it compared. A trailing space, a capital letter or a second space inside a name makes two rows different text, and by default only the first two of those are ignored. The three switches decide it, and the count changes with them — which is the honest version, because there is no setting that is right for every file.

What does choosing key columns actually do?

It declares that two rows with the same email — or the same id, or the same name and city together — are the same row, whatever else differs. That is the powerful setting and the dangerous one: everything outside the key stops being looked at, so a row carrying a phone number nothing else has can be dropped for matching on a column nobody read.

What is a conflict?

A group of rows that matched on the key and differ somewhere else. Those are not duplicates at all — they are two records about one thing, holding different values, and keeping the first is a coin toss dressed up as a rule. They are listed by row number so you can reconcile them by hand, which is the one job this tool deliberately will not do for you.

Why does it list every matched group, not just a count?

Because a deduplicator that answers only with a smaller file gives you no way to check it did the right thing, and the failure you care about — a row that should have stayed — leaves no trace. Every group is listed with its row numbers and which one survived, so the output can be audited rather than trusted.

Does keeping the first or the last matter?

Only when the rows differ, which is exactly when the page marks them as a conflict. For identical rows it makes no difference at all. When it does matter, the last is usually the more recent record and the first is usually the original one — but the file cannot tell you which, so the tool does not pretend to know.

Are the kept rows changed in any way?

No. The values are written back exactly as they were read, including the spaces and the capitals the comparison ignored. Quoting is re-applied where RFC 4180 requires it and dropped where it does not, so a field that was quoted for no reason comes back unquoted — reading the output gives the same values either way.

Why are rows with the wrong number of fields never duplicates?

Because a short or long row is compared as what it is, and it holds a different number of values from every other row, so it can never match one. That is usually a delimiter inside an unquoted field, which shifts everything after it. Those rows are listed so you can fix the quoting first.

[ THE MATHS ]
How many duplicates one file has
Compared rawthe fewest
+ trim the endsmore
+ ignore casemore again
+ collapse spacesthe most
Key columns onlya different question
Each step down finds more duplicates and risks merging people who differ. No setting is right for every file.
[ WHY THE KEY IS NOT JOINED ON A COMMA ]

Join a key on a comma and ["a,b", "c"] and ["a", "b,c"] become the same key — two different rows reported as duplicates, which for a deduplicator means deleting one of them. This uses a character that cannot occur inside a field.

[ NEXT ]
50Line ToolsThe same job for lines of plain text
82CSV Column MaskerBefore the deduplicated file goes anywhere
63JSON ⇄ CSV ConverterIf the export came out as JSON
48Text ComparisonFor differences you cannot see
[ IMPORTANT ]

Nothing you paste is transmitted or stored. Choosing key columns means everything outside them is ignored when deciding what counts as the same row, so a row holding real data can be dropped — read the conflicts before relying on the result. Nothing you type here is sent to our servers — the calculation runs entirely in your browser.