Find Duplicate Lines in Text Online — See What's Repeated

How the duplicate finder works

When you paste text into Flowfiles, the tool builds a frequency map of all lines. Any line that appears more than once is listed in the "Duplicate lines found" panel, along with a count showing how many times it appears in the original input.

This panel updates in real time as you type or paste. Each duplicate entry shows the line content (truncated at 80 characters for readability) and a count badge. Up to 100 unique duplicate lines are shown; if there are more, a summary note is displayed.

Why finding duplicates before removing them matters

Blind deduplication is fine for lists where all entries are equivalent. But sometimes a "duplicate" is actually a sign of a deeper problem:

• An email appearing 50 times in a subscriber list may indicate a form submission bug that created false entries.
• A keyword appearing in every row of a keyword export might be a header row that was not excluded during export.
• A URL appearing twice might indicate that two different pages were mistakenly consolidated, not that there is a genuine duplicate.
• A configuration value appearing multiple times with different surrounding context might indicate a merge conflict artifact.

Seeing the duplicates first helps you decide whether to remove them, investigate them, or leave them intentionally.

Finding duplicates with command-line tools

For large files or automated workflows, the terminal is more appropriate. Here are the standard commands to list duplicate lines:

• List lines that appear more than once (sorted):
  sort input.txt | uniq -d
• Show each line with its count:
  sort input.txt | uniq -c | sort -rn
• Show only unique lines (appearing exactly once):
  sort input.txt | uniq -u
• Case-insensitive duplicate check:
  sort -f input.txt | uniq -di

The browser tool is faster for interactive exploration; the command line is better for scripting or very large files.

Finding duplicates in spreadsheets

In Excel or Google Sheets, you can highlight duplicate values using conditional formatting. But for plain text lists, that approach requires importing the data first. Flowfiles handles it directly without any import step.

• Excel: Home > Conditional Formatting > Highlight Cell Rules > Duplicate Values
• Google Sheets: Format > Conditional formatting > Custom formula: =COUNTIF(A:A,A1)>1

For a column of plain text values, pasting into Flowfiles is significantly faster than setting up conditional formatting rules.

Frequently asked questions