Pythia 12B
Release in the Pythia family · version pythia-12b
Maintained by EleutherAI1
The largest model in EleutherAI's Pythia suite: an English decoder-only language model with about 11.8 billion total parameters, trained on the Pile (not deduplicated) for about 300 billion tokens. EleutherAI publishes 154 checkpoints from initialization to the end of training as Hugging Face branches.12
- Model hub: Model card (Hugging Face) (external site: huggingface.co)
- Repository: Pythia repository (external site: github.com)
- Paper: Pythia paper (arXiv 2304.01373) (external site: arxiv.org)
- Dataset hub: Pre-shuffled, pre-tokenized Pile used for Pythia (external site: huggingface.co)
Availability and license
Overall availability
Downloadable from Hugging Face without gating.1
Availability is separate from permission: read the license before using or redistributing.
Apache License 2.0 (external site: raw.githubusercontent.com)132
Apache License 2.0 (GPT-NeoX) (external site: raw.githubusercontent.com)4
The Pythia repository states its Apache 2.0 license applies to the code and to the Pythia models. The Pile training data has no single license; its components carry their own terms (see the Pile record).26
Model-disclosure tier
Every item in the model checklist is public, including the training data itself, and the weights and code are under OSI-approved licenses.
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| WeightsCan the general public download the model parameters for this release? | Public | Final weights plus 154 intermediate checkpoints (step 0, ten log-spaced early steps, and every 1,000 steps to 143,000) are published as Hugging Face branches.1 |
| Inference codeIs code for running the model published? | Public | The card documents loading the model with Hugging Face Transformers (GPTNeoXForCausalLM).1 |
| Training codeIs the code used to train the model published? | Public | Trained with EleutherAI's GPT-NeoX library; the Pythia repository publishes the GPT-NeoX config files and step-by-step instructions for reproducing training.21 |
| Training-data informationPublic = the training data itself can be obtained. Partial = composition or sources are documented without full access. | Public | EleutherAI hosts the pre-tokenized training data on Hugging Face in the order the models saw it, with a script to reconstruct the dataloader. The Pile itself is documented in a paper and datasheet; its original download channel is discussed on the Pile record.25 |
| Training recipeAre the training configuration and procedure documented in enough detail to follow? | Public | The card and repository give architecture, batch size (about 2M tokens), learning rate, step count, and precision, and the paper's appendix documents the changes from the earlier v0 runs.12 |
| Evaluation materialsPublic = evaluation code or prompts that let others re-run the evaluations are published. Partial = results only. | Public | All Pythia models were evaluated with EleutherAI's LM Evaluation Harness; results by model and step are in the repository's results/json directory.1 |
What it is useful for
Run and use notes
- The card notes that Pythia uses the same tokenizer as GPT-NeoX-20B and that branch step143000 is identical to the main branch.1
Organization context
Provenance and derivatives
Trained from scratch by EleutherAI on the Pile, without deduplication, using the GPT-NeoX library. The current weights come from the April 2023 retraining of the suite; the earlier v0 models remain available separately. The repository notes that the 6.9B and 12B models used a different initialization than intended because the value was not set in the config.12
- Derived from: The Pile — Pretraining data (standard, non-deduplicated version).
Other releases in the Pythia family
No other releases in this family have been assessed.
U.S. eligibility
Eligible · basis: U.S. nonprofit or lab
The model card states the model was developed by EleutherAI, which states it incorporated as a non-profit research institute in early 2023; the IRS exempt-organization extract for the District of Columbia lists EleutherAI Institute in Washington, D.C., as a 501(c)(3) organization.178
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.