The Pile
Project record
Maintained by EleutherAI364
The Pile is an 825 GiB English text dataset for language-model pretraining, assembled by EleutherAI from 22 component datasets that include web text (Pile-CC, from Common Crawl), academic and professional sources such as PubMed Central, arXiv, and FreeLaw, books, GitHub code, and dialogue. It was described in a December 2020 paper and a 2022 datasheet and is the training corpus of EleutherAI's Pythia models. Its current availability is limited; see access conditions.23411
- Website: The Pile website (external site: pile.eleuther.ai)
- Paper: Pile paper (arXiv 2101.00027) (external site: arxiv.org)
- Paper: Datasheet for the Pile (arXiv 2201.07311) (external site: arxiv.org)
- Repository: Replication code (EleutherAI/the-pile) (external site: github.com)
- Dataset hub: Dataset card (Hugging Face) (external site: huggingface.co)
Availability and license
Overall availability
The Pile website and replication-code README still point to a download hosted by the Eye, which the Pythia model card calls a community mirror; when this catalog checked on 2026-09-29, that host presented an expired certificate and the Pile download path returned a not-found error. The Hugging Face "EleutherAI/pile" repository contains only a dataset card and a loading script that downloads from that host. The 2025 Common Pile paper, whose authors include EleutherAI researchers, states that the use of unlicensed training data has resulted in DMCA takedowns of datasets such as the Pile. EleutherAI's Hugging Face organization still hosts a deduplicated text version without a dataset card, the pre-tokenized data used to train Pythia, and the validation and test splits (tagged with an MIT license).141167108129
Availability is separate from permission: read the license before using or redistributing.
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| AccessCan the data be obtained, and on what terms? | Partial | The original download channel could not be verified; EleutherAI hosts a deduplicated version, pre-tokenized versions, and the validation and test splits on Hugging Face.11089 |
| ProvenanceAre the data's origins documented? | Public | The paper, datasheet, and replication-code README list the 22 components with their sources, sizes, and sampling weights, and each document is tagged with its component.2349 |
| DocumentationIs there a datasheet, card, or equivalent documentation? | Public | A paper and a separate datasheet document the dataset.23 |
| LicensingAre the licensing terms stated? | Partial | Licensing is left to each component, and many component licenses are listed as unknown.67 |
| Stated limitationsDoes the documentation state known limitations or risks? | Public | EleutherAI's Pythia card warns that the Pile is known to contain profanity and lewd or otherwise offensive text and points to the Pile paper's discussion of documented gender, religious, and racial biases.11 |
What it is useful for
Organization context
Provenance and derivatives
Assembled by EleutherAI from 22 existing or newly scraped sources, including Pile-CC (text extracted from Common Crawl), PubMed Central, Books3, OpenWebText2, arXiv, GitHub, FreeLaw, Stack Exchange, USPTO backgrounds, Project Gutenberg (PG-19), OpenSubtitles, English Wikipedia, and others. The datasheet describes the sources as a mix of original scrapes, data made available by the data owners, and third-party scrapes.43
- Derived from: Common Crawl (Pile-CC component) — Largest single component by raw size.
Catalog records that name this entry in their provenance:
U.S. eligibility
Eligible · basis: U.S.-governed project
The datasheet describes the Pile as created by EleutherAI, and its website, replication code, and Hugging Face copies are maintained under EleutherAI's domains and organizations. EleutherAI states it incorporated as a non-profit research institute in early 2023, and the IRS exempt-organization extract for the District of Columbia lists EleutherAI Institute in Washington, D.C., as a 501(c)(3) organization.31461314
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.