README.md

---
license: mit
datasets:
- OpenCoder-LLM/fineweb-code-corpus
- OpenCoder-LLM/fineweb-math-corpus
- OpenCoder-LLM/RefineCode-code-corpus-meta
- OpenCoder-LLM/opc-annealing-corpus
language:
- en
- zh
---

<div align="center">
  <img src="https://github.com/OpenCoder-llm/opencoder-llm.github.io/blob/main/static/images/opencoder_icon.jpg?raw=true" width="50%" alt="OpenCoder-Icon" />
</div>



<p align="center">
  <!-- <a href="https://arxiv.org/pdf/2411.04905"><b>Paper Link</b>👁️</a> -->
  🏠 <a href="https://opencoder-llm.github.io/">Home Page</a>&nbsp&nbsp | 
  &nbsp&nbsp 🤗 <a href="https://huggingface.co/collections/infly/opencoder-672cec44bbb86c39910fb55e">Model</a>&nbsp&nbsp | 
  &nbsp&nbsp 📊 <a href="https://huggingface.co/collections/OpenCoder-LLM/opencoder-datasets-672e6db6a0fed24bd69ef1c2">Dataset</a>&nbsp&nbsp | 
  &nbsp&nbsp 📄<a href="https://arxiv.org/abs/2411.04905">Paper</a>&nbsp&nbsp
</p>


## 1. Introduction
**OpenCoder** is an open and reproducible code LLM family which includes 1.5B and 8B base and chat models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is pretrained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, and supervised finetuned on over 4.5M high-quality SFT examples, finally reaching the performance of top-tier code LLMs. 

This repository contains all the intermediate checkpoints of OpenCoder-1.5B-Base, saving in different branches. For the final results, please refer to 🤗 [OpenCoder-1.5B-Base](https://huggingface.co/infly/OpenCoder-1.5B-Base)

## 2. Branches Overview

- `pretrain_iter_0001000` - `pretrain_iter_0484865`: Intermediate checkpoints during the pretraining stage.

- `anneal_iter_0001000` - `anneal_iter_0023841`: Intermediate checkpoints during the annealing stage. 

We use `pretrain_iter_0373000` as the starting point for the annealing stage, and use `anneal_iter_0023000` as the final base model.