GPT-NeoX
Project record
Maintained by EleutherAI21
GPT-NeoX is EleutherAI's library for training large autoregressive language models on GPUs. It builds on NVIDIA's Megatron-LM and on DeepSpeed, and supports distributed training with ZeRO and 3D parallelism, several positional-embedding and attention options, mixture of experts, and predefined configurations for architectures such as Pythia and Llama. EleutherAI used it to train GPT-NeoX-20B and the Pythia suite.2
- Repository: Repository (external site: github.com)
- License: LICENSE (external site: raw.githubusercontent.com)
- Release notes: Release v2.0 (external site: github.com)
Availability and license
Overall availability
Source code on GitHub under the Apache License 2.0; installed from the repository with pip requirements files.132
Availability is separate from permission: read the license before using or redistributing.
The README states the code is based on NVIDIA code licensed under Apache 2.0, that modified NVIDIA files keep NVIDIA copyright headers, and that some code from other authors is marked with its own license. After the Apache 2.0 text, the LICENSE file also reproduces a BSD-style license for NVIDIA code, Apache 2.0 for Hugging Face and Google Research code, and the MIT license for code from Facebook's Fairseq.23
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| Source codeIs the source code publicly readable? | Public | 1 |
| DocumentationIs user documentation published? | Public | The README covers environment setup, configuration, datasets, training, inference, evaluation, exporting to Hugging Face format, monitoring, and multi-node launching.2 |
| Training codeDoes it include code for training models? | Public | The library is itself training code for large-scale language model pretraining and fine-tuning, including DPO, KTO, and reward modeling.2 |
| Data informationAre the data it expects or ships with documented? | Public | The README documents preconfigured datasets (most components of the Pile, and the Pile train set) and how to tokenize custom data into the library's .bin/.idx format.2 |
| ReproducibilityAre instructions for reproducing reported results published? | Public | Versioned releases pin earlier states: v1.0 keeps the stack used for GPT-NeoX-20B and Pythia, and v2.0 moved to upstream DeepSpeed. The Pythia repository gives step-by-step instructions for reproducing Pythia training with GPT-NeoX.24 |
What it is useful for
Training models with billions of parameters from scratch on GPU clusters, including preparing tokenized datasets, launching on Slurm or MPI, and evaluating with the LM Evaluation Harness. The README recommends Hugging Face Transformers instead for general inference.2
Run and use notes
- The README states the codebase was primarily developed and tested with Python 3.8 to 3.10 and PyTorch 1.8 to 2.0, relies on EleutherAI's DeeperSpeed fork of DeepSpeed, and supports AMD MI100 and MI250X GPUs through JIT-compiled fused kernels.2
Organization context
Provenance and derivatives
Based on NVIDIA's Megatron-LM code and augmented with techniques from DeepSpeed, per the README.2
- Derived from: Megatron-LM (NVIDIA) (external site: github.com) — Original code base.
- Derived from: DeepSpeed (external site: github.com) — Used through EleutherAI's DeeperSpeed fork.
U.S. eligibility
Eligible · basis: U.S.-governed project
GPT-NeoX is maintained in EleutherAI's GitHub organization, and the README states that files without an NVIDIA header are the exclusive copyright of EleutherAI. EleutherAI states it incorporated as a non-profit research institute in early 2023, and the IRS exempt-organization extract for the District of Columbia lists EleutherAI Institute in Washington, D.C., as a 501(c)(3) organization.1256
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.