Model release

Pythia 12B

Pythia 12B is an open-weight model release in the Pythia family from EleutherAI1, version pythia-12b, released Apr 3, 2023. Its weights can be downloaded by the public1 under the Apache License 2.01 3 2.

The largest model in EleutherAI's Pythia suite: an English decoder-only language model with about 11.8 billion total parameters, trained on the Pile (not deduplicated) for about 300 billion tokens. EleutherAI publishes 154 checkpoints from initialization to the end of training as Hugging Face branches.1 2

Model hub: Model card (Hugging Face) (external site: huggingface.co)
Repository: Pythia repository (external site: github.com)
Paper: Pythia paper (arXiv 2304.01373) (external site: arxiv.org)
Dataset hub: Pre-shuffled, pre-tokenized Pile used for Pythia (external site: huggingface.co)

On AI on Hugging Face: EleutherAI/pythia-12b, EleutherAI/pile-standard-pythia-preshuffled

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Apr 3, 2023

Availability and license

Overall availability

Public

Downloadable from Hugging Face without gating.1

Availability fact review: not separately recorded

Availability is separate from permission: read the license before using or redistributing.

Apache License 2.0 (external site: raw.githubusercontent.com)1 3 2
Applies to: Weights and code · SPDX Apache-2.0 · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the Apache License 2.0
Apache License 2.0 (GPT-NeoX) (external site: raw.githubusercontent.com)4
Applies to: Code · SPDX Apache-2.0 · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the Apache License 2.0

The Pythia repository states its Apache 2.0 license applies to the code and to the Pythia models. The Pile training data has no single license; its components carry their own terms (see the Pile record).2 6

Component reuse rights

A readable or downloadable component is not automatically reusable. These indicators concern recorded license evidence, not system certification.

weights: Unknown — no complete fact-level rights review
code: Unknown — no complete fact-level rights review
data: Unknown — no complete fact-level rights review
documentation: Unknown — no complete fact-level rights review

No complete system-rights review is recorded for this release.

Model-disclosure tier

Computed from the checklist below using USASI rubric v0.2. An editorial category, not a certification.

Open-weight

The model parameters for this release can be downloaded by the public. License terms may still restrict use, redistribution, or commercial use.

How tiers are computed

Public materials checklist

Items for a model under USASI rubric v0.2. Unknown means unassessed or insufficient evidence.

Public materials checklist for Pythia 12B
Item	Status	Notes and evidence
WeightsCan the general public download the model parameters for this release?	Public	Final weights plus 154 intermediate checkpoints (step 0, ten log-spaced early steps, and every 1,000 steps to 143,000) are published as Hugging Face branches.1
Inference codeIs code for running the model published?	Public	The card documents loading the model with Hugging Face Transformers (GPTNeoXForCausalLM).1
Training codeIs the code used to train the model published?	Public	Trained with EleutherAI's GPT-NeoX library; the Pythia repository publishes the GPT-NeoX config files and step-by-step instructions for reproducing training.2 1
Training-data informationDoes the information cover provenance, scope, acquisition, selection, labeling, processing, and where data or alternatives can be obtained? Access alone does not establish completeness.	Unknown	Not assessed.
Training-data accessCan the training data be obtained? This is independent of information completeness and reuse rights; original unshareable data need not be downloadable.	Unknown	Not assessed.
Complete training pipelineIs the complete base-training and preprocessing pipeline published, including configuration? Fine-tuning code or an inference SDK alone is insufficient.	Unknown	Not assessed.
Legacy data assessment (v0.1)Historical assessment combining download access and disclosure. Preserved for traceability; excluded from the v0.2 tier calculation. See the new separate assessments above.	Public	EleutherAI hosts the pre-tokenized training data on Hugging Face in the order the models saw it, with a script to reconstruct the dataloader. The Pile itself is documented in a paper and datasheet; its original download channel is discussed on the Pile record.2 5
Training recipeAre the training configuration and procedure documented in enough detail to follow?	Public	The card and repository give architecture, batch size (about 2M tokens), learning rate, step count, and precision, and the paper's appendix documents the changes from the earlier v0 runs.1 2
Evaluation materialsPublic = evaluation code or prompts that let others re-run the evaluations are published. Partial = results only.	Public	All Pythia models were evaluated with EleutherAI's LM Evaluation Harness; results by model and step are in the repository's results/json directory.1

What it is useful for

Research on the behavior, functionality, and limitations of large language models, such as interpretability, memorization, and learning dynamics. The card states it is not intended for deployment or human-facing use and has not been instruction-tuned.1 2

Run and use notes

Documented facts only. No hardware or performance claims are made without a cited source and stated assumptions.

The card notes that Pythia uses the same tokenizer as GPT-NeoX-20B and that branch step143000 is identical to the main branch.1

Organization context

EleutherAI

Provenance and derivatives

Trained from scratch by EleutherAI on the Pile, without deduplication, using the GPT-NeoX library. The current weights come from the April 2023 retraining of the suite; the earlier v0 models remain available separately. The repository notes that the 6.9B and 12B models used a different initialization than intended because the value was not set in the config.1 2

Derived from: The Pile — Pretraining data (standard, non-deduplicated version).

Other releases in the Pythia family

No other releases in this family have been assessed.

Pythia family overview

What this catalog does not know

Unknown means the sources reviewed for this record do not document it. It is not evidence that something does not exist.

Training-data information: unknown.
Training-data access: unknown.
Complete training pipeline: unknown.

Have a primary source? How to report a correction.

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S. nonprofit or lab

The model card states the model was developed by EleutherAI, which states it incorporated as a non-profit research institute in early 2023; the IRS exempt-organization extract for the District of Columbia lists EleutherAI Institute in Washington, D.C., as a 501(c)(3) organization.1 7 8

Assessed Sep 29, 2026

Sources

1.
EleutherAI/pythia-12b model card (external site: huggingface.co)
EleutherAI · Model card · accessed Sep 29, 2026
2.
EleutherAI/pythia: Interpreting Transformers Across Time and Scale (external site: github.com)
EleutherAI · Repository · accessed Sep 29, 2026
3.
EleutherAI/pythia LICENSE (external site: raw.githubusercontent.com)
EleutherAI · License · accessed Sep 29, 2026
4.
EleutherAI/gpt-neox LICENSE (external site: raw.githubusercontent.com)
EleutherAI · License · accessed Sep 29, 2026
5.
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (arXiv 2304.01373) (external site: arxiv.org)
EleutherAI (arXiv) · Paper · published Apr 3, 2023 · accessed Sep 29, 2026
6.
EleutherAI/pile dataset card (external site: huggingface.co)
EleutherAI · Dataset card · accessed Sep 29, 2026
7.
About | EleutherAI (external site: eleuther.ai)
EleutherAI · Official page · accessed Sep 29, 2026
8.
IRS Exempt Organizations Business Master File Extract (District of Columbia) (external site: irs.gov)
U.S. Internal Revenue Service · Filing · accessed Sep 29, 2026

Email a correction Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.

Cite this page

APA style

USASI. (2026, September 29). Pythia 12B. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/pythia-12b/

BibTeX

@misc{usasi_pythia_12b,
  author = {{USASI}},
  title = {Pythia 12B},
  year = {2026},
  month = sep,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/pythia-12b/}},
  note = {United States of America Superintelligence. Last updated 2026-09-29}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

For maintainers: a “Listed in USASI” badge

Links to this page. It says the entry is listed, not endorsed or rated. About the badge

Markdown (for a README)

[![Listed in the USASI catalog](https://unitedstatesofamericasuperintelligence.com/badges/listed.svg)](https://unitedstatesofamericasuperintelligence.com/open/pythia-12b/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/open/pythia-12b/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>