DatasetDataset

RedPajama

RedPajama is a dataset from Together AI (Together Computer, Inc.)1 3 7, version V2, released Oct 30, 2023. The data is publicly available1 2 under the Apache License 2.0 (dataset loading and processing code)4 1.

RedPajama is Together AI's open pretraining dataset project. RedPajama-V1 (RedPajama-Data-1T, 2023) is a 1.2-trillion-token reproduction of the LLaMA training-data recipe drawn from Common Crawl, C4, GitHub, arXiv, Wikipedia, and Stack Exchange. The current version, RedPajama-V2 (October 2023), contains over 100 billion documents from 84 Common Crawl snapshots in English, German, French, Spanish, and Italian, with more than 40 precomputed quality signals and duplicate markers; its deduplicated, annotated portion is about 30 trillion tokens.2 1 3 5 6

Dataset hub: RedPajama-Data-V2 (Hugging Face) (external site: huggingface.co)
Dataset hub: RedPajama-Data-1T (Hugging Face) (external site: huggingface.co)
Repository: RedPajama-Data code repository (external site: github.com)
Paper: RedPajama paper (arXiv 2411.12372) (external site: arxiv.org)
Release notes: RedPajama-Data-v2 announcement (external site: together.ai)

On AI on Hugging Face: togethercomputer/RedPajama-Data-1T, togethercomputer/RedPajama-Data-V2

Last reviewedSep 29, 2026Entry updated Sep 29, 2026Documented release Oct 30, 2023

Availability and license

Overall availability

Public

Both versions are downloadable from Hugging Face without gating. The V2 card directs users to the Common Crawl Foundation Terms of Use for the data. In V1 the 'book' configuration is marked defunct and no longer accessible because of reported copyright infringement in its Books3 content.1 2

Availability fact review: not separately recorded

Availability is separate from permission: read the license before using or redistributing.

Apache License 2.0 (dataset loading and processing code) (external site: raw.githubusercontent.com)4 1
Applies to: Code · SPDX Apache-2.0 · on the rubric’s OSI-approved software-license list · fact-level rights review not recorded
Plain-language guide to the Apache License 2.0

Neither version grants a single license for the data. The V2 card refers users to the Common Crawl Foundation Terms of Use for the data and applies Apache 2.0 to the code; the V1 card asks users to follow the license of each subset (for example, the GitHub subset was limited to MIT, BSD, and Apache-licensed projects).1 2

Component reuse rights

A readable or downloadable component is not automatically reusable. These indicators concern recorded license evidence, not system certification.

weights: Unknown — no complete fact-level rights review
code: Unknown — no complete fact-level rights review
data: Unknown — no complete fact-level rights review
documentation: Unknown — no complete fact-level rights review

No complete system-rights review is recorded for this release.

Public materials checklist

Items for a dataset under USASI rubric v0.2. Unknown means unassessed or insufficient evidence.

Public materials checklist for RedPajama
Item	Status	Notes and evidence
AccessCan the data be obtained, and on what terms?	Public	V2 and V1 are downloadable without gating; V1's book subset is no longer accessible.1 2
ProvenanceAre the data's origins documented?	Public	V2 documents carry their source URL, domain, and Common Crawl snapshot ID, and the card describes the CCNet processing; the V1 card lists each source with its token count and processing steps.1 2
DocumentationIs there a datasheet, card, or equivalent documentation?	Public	Dataset cards, an announcement post, a NeurIPS 2024 Datasets and Benchmarks paper, and the full pipeline code for recreating V2 (including quality signals) are published.1 5 6 3
LicensingAre the licensing terms stated?	Partial	Code is Apache 2.0; data terms defer to Common Crawl's terms (V2) or to each subset's license (V1).1 2
Stated limitationsDoes the documentation state known limitations or risks?	Unknown	Not assessed.

What it is useful for

Building filtered pretraining mixes for large language models: users select documents by snapshot, language, and partition and apply the provided quality signals and duplicate IDs to create their own subsets.1 6

Run and use notes

Documented facts only. No hardware or performance claims are made without a cited source and stated assumptions.

The V2 card notes that downloading a full snapshot for a given partition and language requires about 1 TB of disk space per snapshot, and provides a small 'sample' configuration for exploration.1

Organization context

Together AI

Provenance and derivatives

V2 was built from 84 Common Crawl snapshots processed with the CCNet pipeline into head, middle, and tail perplexity buckets; head and middle documents were annotated with quality signals and duplicates were marked with a Bloom filter rather than removed. V1 followed the LLaMA paper's data recipe using Common Crawl dumps processed with CCNet, C4, permissively licensed GitHub code, arXiv, Wikipedia, and Stack Exchange.1 2

Derived from: Common Crawl — Sole source for V2 and the largest source for V1.

What this catalog does not know

Unknown means the sources reviewed for this record do not document it. It is not evidence that something does not exist.

Stated limitations: unknown.

Have a primary source? How to report a correction.

U.S. eligibility

Project eligibility rests on documented governing or maintaining entities, not on contributors.

Eligible · basis: U.S.-governed project

The datasets are published under Together's Hugging Face and GitHub organizations, and the dataset card's software citation names Together Computer as author. Together's terms of service identify Together Computer, Inc., a Delaware corporation, and give a San Francisco, California address for copyright notices. The V2 card also thanks the RedPajama-V1 partners (including Stanford research groups, Mila, Université de Montréal, ETH DS3Lab, LAION, and Ontocord.ai); Together remains the publishing and maintaining entity.1 3 7

Assessed Sep 29, 2026

Sources

1.
togethercomputer/RedPajama-Data-V2 dataset card (external site: huggingface.co)
Together AI · Dataset card · accessed Sep 29, 2026
2.
togethercomputer/RedPajama-Data-1T dataset card (external site: huggingface.co)
Together AI · Dataset card · accessed Sep 29, 2026
3.
RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models (repository README) (external site: raw.githubusercontent.com)
Together AI · Repository · accessed Sep 29, 2026
4.
togethercomputer/RedPajama-Data LICENSE (external site: raw.githubusercontent.com)
Together AI · License · accessed Sep 29, 2026
5.
RedPajama-Data-v2: An open dataset with 30 trillion tokens for training large language models (external site: together.ai)
Together AI · Announcement · published Oct 30, 2023 · accessed Sep 29, 2026
6.
RedPajama: an Open Dataset for Training Large Language Models (arXiv 2411.12372) (external site: arxiv.org)
Together AI and collaborators (arXiv) · Paper · published Nov 19, 2024 · accessed Sep 29, 2026
7.
Terms of Service | Together AI (external site: together.ai)
Together AI · Official page · accessed Sep 29, 2026

Email a correction Report a correction Edit this entry

This listing is not an endorsement, a safety assessment, or a federal approval.

Cite this page

APA style

USASI. (2026, September 29). RedPajama. United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/redpajama/

BibTeX

@misc{usasi_redpajama,
  author = {{USASI}},
  title = {RedPajama},
  year = {2026},
  month = sep,
  howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/redpajama/}},
  note = {United States of America Superintelligence. Last updated 2026-09-29}
}

The date is when this page was last updated. Add the date you read it if your style needs one.

For maintainers: a “Listed in USASI” badge

Links to this page. It says the entry is listed, not endorsed or rated. About the badge

Markdown (for a README)

[![Listed in the USASI catalog](https://unitedstatesofamericasuperintelligence.com/badges/listed.svg)](https://unitedstatesofamericasuperintelligence.com/open/redpajama/)

HTML

<a href="https://unitedstatesofamericasuperintelligence.com/open/redpajama/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>