tokenizers
by huggingface (organization)
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production (description from the project’s GitHub page)
Repository facts
- Stars
- 11,183
- Forks
- 1,226
- Open issues and pull requests
- 198
- Primary language
- Rust
- License
- Apache License 2.0 (Apache-2.0)
- Owner
- huggingface · Organization
- Created
- Nov 1, 2019
- Last push
- Oct 18, 2026
Topics
Add this badge to your README
Shows this repository’s rank in AI on GitHub and links back to this page. Ranks are by stars and change when the list is refreshed. The badge is not an endorsement or a rating. About the badges
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/github/huggingface--tokenizers/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/github/huggingface--tokenizers/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/github/huggingface--tokenizers.svg" alt="Ranked #716 in AI on GitHub" height="20"></a>
Related projects
Projects that share this one’s GitHub topics.
Rank 4,789: explosion/spacy-transformers
🛸 Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy
1,413 starsPythonMITLast push Mar 27, 2026
Rank 4,772: SKTBrain/KoBERT
Korean BERT pre-trained cased (KoBERT)
1,419 starsPythonApache-2.0Last push Jun 14, 2025
Rank 7,577: ymcui/MacBERT
Revisiting Pre-trained Models for Chinese Natural Language Processing (MacBERT)
722 starsApache-2.0Last push Apr 19, 2026
Rank 3,890: nlpodyssey/spago
Self-contained Machine Learning and Natural Language Processing library in Go
1,852 starsGoBSD-2-ClauseLast push Apr 1, 2025
Rank 7,463: primeqa/primeqa
The prime repository for state-of-the-art Multilingual Question Answering research and development.
739 starsPythonApache-2.0Last push Sep 18, 2025
Rank 998: jessevig/bertviz
BertViz: Visualize Attention in Transformer Models
8,200 starsPythonApache-2.0Last push Jan 8, 2026