The duplicate remover splits text by a delimiter you choose, removes identical segments, and reports which items were duplicated and how many times each appeared. It is the fastest way to deduplicate email lists, keyword banks, log lines, or any text where repeated items need to go.
Exact Match Only — No Fuzzy Deduplication
Deduplication is byte-for-byte exact matching. These pairs are treated as distinct by default:
Appleandapple— case difference, not mergedhelloandhello— trailing space matters，(full-width comma) and,(half-width comma) — treated as different characters
This is intentional: fuzzy matching would silently merge items that carry different meaning in real data. If you need case-insensitive deduplication, convert all text to one case first.
Delimiter Syntax
The delimiter field accepts escape sequences. Common values:
\n— deduplicate line by line (default, works for most lists),— comma-separated values\t— tab-separated (TSV data);— semicolon-separated||or|— any multi-character string works as a delimiter
If the text itself contains the same character as your delimiter, it will be split incorrectly. Choose a delimiter you are confident does not appear inside individual items.
Order and Statistics
The result preserves the first occurrence of each unique item in its original position; all subsequent duplicates are removed. The duplicate statistics list below the output shows each removed value and its total count — useful for spotting the most repeated items in a list or identifying data quality issues.
Output Examples
Deduplicating a line-by-line list
Input with \n delimiter:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Output:
[email protected]
[email protected]
[email protected]
Duplicates found: [email protected] (2), [email protected] (2)
Deduplicating a comma-separated string
Input with , delimiter: red,blue,green,red,yellow,blue
Output: red,blue,green,yellow