text-dedup
by ChenghaoMou (personal account)
All-in-one text de-duplication (description from the project’s GitHub page)
Repository facts
- Stars
- 764
- Forks
- 81
- Open issues and pull requests
- 1
- Primary language
- Python
- License
- Apache License 2.0 (Apache-2.0)
- Owner
- ChenghaoMou · Personal account
- Created
- Mar 13, 2021
- Last push
- Mar 9, 2026
Topics
Add this badge to your README
Shows this repository’s rank in AI on GitHub and links back to this page. Ranks are by stars and change when the list is refreshed. The badge is not an endorsement or a rating. About the badges
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/github/chenghaomou--text-dedup/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/github/chenghaomou--text-dedup/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/github/chenghaomou--text-dedup.svg" alt="Ranked #7,290 in AI on GitHub" height="20"></a>
Related projects
Projects that share this one’s GitHub topics.
Rank 2,281: lynote-ai/humanize-text
Open-source pipeline for rewriting AI-generated text into natural human prose. Need it in your app? Try the Lynote Humanizer API → lynote.ai/ai-humanizer-api
3,448 starsPythonMITLast push Oct 13, 2026
Rank 7,928: cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation…
673 starsPythonMITLast push Jun 2, 2025
Rank 4,472: allenai/dolma
Data and tools for generating and inspecting OLMo pre-training data.
1,559 starsPythonApache-2.0Last push Aug 24, 2026
Rank 4,909: pemistahl/lingua-go
The most accurate natural language detection library for Go, suitable for short text and mixed-language text
1,367 starsGoApache-2.0Last push Feb 6, 2025
Rank 7,768: abadojack/whatlanggo
Natural language detection library for Go
695 starsGoMITLast push Mar 28, 2023