Independent project. Not a U.S. government website.

Support our project (opens Buy Me a Coffee)
USASI

Jump to

Start hereFour ways into the catalogCompanies & LabsU.S. AI organizationsOpen Models & ToolsOpen models, software, datasets, and evaluation toolsLearnExplainers and learning pathsFor kidsAI and Super Intelligence, explained for readers 8 to 12Latest newsSourced news itemsPolicy trackerFederal AI laws, orders, and standards

Generated sky blue circuit-trace artwork for marcelroed/gigatoken, labeled USASI · AI on GitHub, with Personal account · Rust · MIT.
Rank #1,922 4,129 starsRust

gigatoken

by marcelroed (personal account)

Language model tokenization at GB/s (description from the project’s GitHub page)

View on GitHub (external site: github.com)

Repository facts

Stars
4,129
Forks
236
Open issues and pull requests
20
Primary language
Rust
License
MIT License (MIT)
Owner
marcelroed · Personal account
Created
Nov 10, 2025
Last push
Sep 2, 2026

Topics

Rank #1,922 of 9,507 in AI on GitHub, by stars. Data from the GitHub API, fetched Oct 18, 2026; star counts change daily. Stars measure popularity, not quality or safety, and a listing is not an endorsement. How this list is made

Add this badge to your README

Ranked #1,922 in AI on GitHubShows this repository’s rank in AI on GitHub and links back to this page. Ranks are by stars and change when the list is refreshed. The badge is not an endorsement or a rating. About the badges

Markdown (for a README)

[![Ranked #1,922 in AI on GitHub](https://unitedstatesofamericasuperintelligence.com/badges/github/marcelroed--gigatoken.svg)](https://unitedstatesofamericasuperintelligence.com/github/marcelroed--gigatoken/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/github/marcelroed--gigatoken/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/github/marcelroed--gigatoken.svg" alt="Ranked #1,922 in AI on GitHub" height="20"></a>

Related projects

Projects that share this one’s GitHub topics.

  • Rank 7,928: cbaziotis/ekphrasis

    Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation…

    673 starsPythonMITLast push Jun 2, 2025

  • Rank 216: explosion/spaCy

    💫 Industrial-strength Natural Language Processing (NLP) in Python

    33,968 starsPythonMITLast push Oct 12, 2026

  • Rank 6,723: explosion/spacy-streamlit

    👑 spaCy building blocks and visualizers for Streamlit apps

    863 starsPythonMITLast push Jul 29, 2024

  • Rank 7,103: nlp-uoregon/trankit

    Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

    798 starsPythonApache-2.0Last push Jul 22, 2025

  • Rank 9,177: yooper/php-text-analysis

    PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

    534 starsPHPMITLast push Dec 28, 2024

Support Us

Help keep USASI useful.

Leave a tip on Buy Me a Coffee (external site)

Optional. No USASI account required. Payment takes place on the linked provider’s website (Buy Me a Coffee).

About supporting this project