Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

I hope this message finds you well. I've been thoroughly impressed by the rigor and innovation demonstrated in your work on the Nemotron-CC dataset. As I explore this valuable resource, I would be grateful for some clarification regarding two key aspects of the dataset construction:

1.Dataset Composition Questions:
For the 15TB dataset token dataset used in Nemotron-CC's performance evaluation, it is stated to consist of:
7.2TB from Nemotron-CC's own dataset
7.8TB from fixed-ratio mixed professional datasets (math, code, papers, books, patents, Wikipedia, etc.)

Could you please provide details on:
a) How the original 6.3TB Nemotron-CC dataset was expanded to 7.2TB?
b) The exact composition of the 7.8TB mixed dataset - specifically which datasets were used for each category (math, code, papers, etc.) and their respective proportions?

2.Deduplication Process:
For Nemotron-CC's core 6.3TB dataset, could you clarify:
a) What deduplication methods were employed (e.g., exact matching, fuzzy matching, semantic deduplication)?
b) Whether temporal deduplication was performed? For example, if a webpage with the same URL (or identical content) is crawled in multiple years (e.g., 2013, 2014, 2015) , how is this handled?

Thank you for your time and for making this valuable resource available!

Hi AshleyLL! Thank you for your interest and questions. We have added some clarifications to the v2 version of the paper on arXiv: https://arxiv.org/abs/2412.02595 .

1.(a) For the 15T token training run, a two-phase curriculum was employed that is described in more detail in https://arxiv.org/abs/2412.15285v1 . The first phase of 9T tokens used 59% English Common Crawl data (5.31T) and the second phase of 6T tokens used 31% (1.86T), for a combined total of 47.8% (7.17T). In the first phase, we used medium, medium-high, and high
quality data (real and synthetic), and in the second phase we used only high quality data (real and synthetic). The weights were a bit different for different quality buckets. Generally speaking, for benchmarks we found using about 4-8 epochs of high quality data is best before it becomes more beneficial to start using tokens from medium-high and medium quality buckets. That said, we also wanted to make sure we have diverse long-tail data so we included more medium-quality data than is optimal from purely a benchmark perspective. The dataset is available broken down by buckets and synthetic data type, so we encourage the community to do their own blending and curriculum experiments.

Please also see the Nemotron-H paper, though that used an expanded 3 or 4 phase curriculum: https://arxiv.org/abs/2504.03624

1.(b) The 27% non-English-Common-Crawl portion includes a lot of internal datasets we cannot spell out here. But the category breakdown is roughly: Books and patents (9%), papers (9%), code (5%), conversational (3%), wikipedia (1%). (See Table 12 in Appendix D of the updated paper.)

2.(a) We performed global exact and fuzzy deduplication across all 99 Common Crawl snapshots used. Additionally, we split each chunk into 8 roughly equal parts and did exact substring deduplication on each chunk.

2.(b) We did not do any temporal deduplication. If the same URL occurred in multiple years, then there could be multiple copies in the dataset. That said, if the content did not substantially change over the years, then most copies would probably get removed by the global exact and/or fuzzy deduplication from 2.(a).

Hi mkliegl-nv,

Thanks for your excellent work!

I have the same confusion: where does the 15T data come from?

I just cannot see the numbers add up.

Could you please nail down on the numbers and show the breakdown and sources of the 15T data?

I'm sorry if this is confusing. Could you elaborate on what you mean by "I just cannot see the numbers add up?" I suspect there may be some basic misunderstanding. For example, we are not talking about 15T tokens of unique data here. We are talking about a long-horizon training run for 15T tokens. Some of the higher quality datasets will be trained on for multiple epochs, as is common practice. Please see the references I linked to, too. Hope this helps.

Thanks a lot Mkliegl-nv!
Sorry for my confusion. I have been trying to figure out the cross linkage of your papers.
Lets start from the beginning:

1. How did the original 6.3TB Nemotron-CC dataset got expanded into 7.2TB?
2. Then, 15TB - 7.2TB = 7.8TB, where is this data?
   I cannot seem to match within the other two papers.
3. I have seen your update in the latest version of the Nemotron-CC paper. But that directly jumped to 9T, without explaining its linkage to the original 6.3TB and 7.2TB of Nemotron-CC dataset.
   How is the 9T related to 6.3TB and 7.2TB?
4. It is okay to have 4-8 epochs of high quality data, but which sub-portion of the dataset and how much is it? 200GB? 500GB? 1TB?
   Thanks a lot!

So I think, basically, you're asking for the exact datasets, blend and curriculum used for the 15T token run. That's a valid question but beyond the scope of this paper and what I can share. As mentioned, various internal datasets were used, but this paper was focused only on English Common Crawl data which we have publicly released. For the Crawl portion, I have given the general idea above in 1(a), the key point being 4-8 epochs of high quality Nemotron-CC data (real and synthetic - e.g., 2 epochs of real HQ + 2 epochs of synthetic diverse QA HQ + 1 epoch of synthetic distill HQ would count as 2 epochs of real + 3 epochs of synthetic). In any case the curriculum used for that model has been superseded. See, e.g., the curriculum described in the Nemotron-H paper, Figure 4, for a more recent high-level curriculum used. The Nemotron-H models have also been released publicly for research purposes.

Hi mkliegl-nv,
Thanks so much for your prompt reply!
I am not asking for the exact content of the dataset.
I just try to replicate the results in your paper as close as possible.
Please feel free to withhold some information that you cannot share, which I fully understand.
Could you please:

1. help me make the numbers add up (at least), starting from the 6.3T and 7.2T;
2. point me to how to replicate the results in your paper as close as possible.
   Thanks a lot again for your kind help!

Here's the exact Nemotron-CC subset breakdown for the 8B-15T run described in the Nemotron-CC paper. So roughly 5 epochs of HQ real data and 5.8 epochs of HQ synthetic data of various types in total.

Great! Thanks a lot for your help miliegl-nv!

Thank you for your earlier clarification regarding the Nemotron-CC dataset — it's an extremely valuable resource.

I have a couple of follow-up questions regarding the training curriculum:

The table you kindly provided outlines various subsets (e.g., HQ-real, HQ-extract_knowledge) used in different phases of pretraining. However, the token counts mentioned do not seem to align with the total tokens available in the publicly available Nemotron CC dataset. Could you clarify how the categories in the publicly available Nemotron CC dataset were processed to form these specific subsets? Specifically:
Does the publicly available Nemotron CC dataset include explicit quality labels (e.g., "High," "Medium-High," "Medium") for each sample?
Are there data type labels distinguishing real vs. synthetic data? If synthetic, are subtypes like "Distill" or "Diverse QA Pairs" specified?
Were certain categories filtered out, upsampled, downsampled, or reweighted to achieve the token counts shown in the table?

Thank you in advance for any insights — your input would be greatly appreciated.

Based on the interpretation of your earlier responses — that the non-Nemotron-CC portion follows an approximate distribution of Books & Patents : Papers : Code : Conversational Data : Wikipedia in the ratio 9 : 9 : 5 : 3 : 1 — a proxy dataset has been constructed to reflect this proportion.

While the relative proportions of the different data categories remain consistent across the two pre-training stages, the total number of tokens used differs significantly between them. Could you please clarify whether this difference arises from different processing or weighting applied to the same underlying dataset, or whether distinct datasets were used in each stage?

It would be helpful to hear your perspective on whether specific choices of data sources used and their proportional distribution are broadly aligned with what might be considered effective for such a training corpus.

Here's the exact Nemotron-CC subset breakdown for the 8B-15T run described in the Nemotron-CC paper. So roughly 5 epochs of HQ real data and 5.8 epochs of HQ synthetic data of various types in total.

HI mkliegl-ny,

I appreciate your insights, mkliegl-nv.

I'm trying to better understand the dataset progression across your Nemotron research. The documentation shows some variations in reported dataset sizes that I'd like to clarify:

1. Could you explain how the Nemotron-CC dataset grew from the initially reported 6.3T token to 7.2T token?

From the table you gave, I have tried to do the reconciliation but failed, maybe I am missing something here?

2. Is the 7.2T token the one labelled "7169805149903" in your table?

If this could be confirmed, then we will be able to understand the basic starting point.

3. In referencing the 15T token total training data, I notice a gap (15T token - 7.2T token = 7.8T token) that isn't clearly attributed in the publications. Where does this additional data originate from?

I'm not seeking confidential dataset details, but rather trying to understand the methodology sufficiently to align my research with yours and also replicate your work as much as possible. Any clarification you can provide would be valuable for my replication efforts.

Thank you for your consideration.

1. Copy&pasting from one of my other responses above. "Token counts match up for the tokenizer we used (same as the Nemotron-H tokenizer, you can find it on HuggingFace). If you used a different tokenizer, things can of course look a bit different. For example, you will find in the paper (Table 2) that we have 553B high-quality real tokens in the dataset, and in the table above 2.743T total "HQ-real" tokens were used => 2.743T / 553B = 4.96 epochs, or 5x upsampling if you prefer. "HQ-real" is the "contrib/Nemotron/Nemotron-CC/data-jsonl/quality=high/kind=actual/kind2=actual/" part of the public dataset."
2. Yes.
3. The remaining 7.8T tokens are from datasets other than Nemotron-CC. You can see the curriculum and Nemotron-H papers to get some sense of the categories.