pico-decoder-large-1 trained to 125k steps

README.md

@@ -0,0 +1,54 @@
+---
+datasets:
+- pico-lm/pretokenized-dolma
+language:
+- en
+license: apache-2.0
+metrics:
+- pico-lm/perplexity
+pipeline_tag: text-generation
+---
+
+# Pico Decoder Large
+
+**pico-decoder-large** is the largest model (570M) in the current `pico-decoder` suite. It is a full-scale research model designed for in-depth interpretability studies of transformer learning. Trained with [`pico-train`](https://github.com/pico-lm) and fully compatible with [`pico-analyze`](https://github.com/pico-lm), it offers rich checkpointing and analytical insight into large-scale LM behavior.
+
+> NOTE: The `pico-decoder-large-1` branch contains the full commit history for the training run. 
+
+## 🔧 Model Details
+
+| Field               | Value                              |
+|---------------------|------------------------------------|
+| **Architecture**     | Decoder-only transformer (LLaMA-style) |
+| **Parameters**       | 570M                              |
+| **Layers**           | 12                                |
+| **Hidden Size**      | 1536                              |
+| **Feed Forward Size**| 6144                              |
+| **Attention Heads**  | 12                                |
+| **Key/Value Heads**  | 4                                |
+
+## 📚 Training
+
+- **Dataset**: [`pretokenized-dolma`](https://github.com/pico-lm)
+- **Training steps**: 200,000
+- **Batch size**: 1024
+- **Sequence length**: 2048
+- **Optimizer**: AdamW
+- **Learning rate schedule**: Linear decay with warmup
+- **Compute**: 16 A100-SXM4-80GB GPUs
+
+## 📈 Evaluation and Analysis
+
+This model supports fine-grained analysis using [pico-analyze](https://github.com/pico-lm). This tool enables researchers to understand how learning unfolds over training, even at very small scales.
+
+We also evaluate perplexity of the model on the [pico-paloma-tinsy](https://huggingface.co/datasets/pico-lm/pretokenized-paloma-tinsy) dataset.
+
+## 📄 Citation
+
+```bibtex
+@software{pico2025,
+    author = {Diehl Martinez, Richard},
+    title = {Pico: A Lightweight Framework for Studying Language Model Learning Dynamics},
+    year = {2025},
+    url = {https://github.com/pico-lm}
+}

config.json

@@ -0,0 +1,22 @@
+{
+  "activation_hidden_dim": 6144,
+  "architectures": [
+    "PicoDecoderHF"
+  ],
+  "attention_n_heads": 12,
+  "attention_n_kv_heads": 4,
+  "auto_map": {
+    "AutoConfig": "pico_decoder.PicoDecoderHFConfig",
+    "AutoModelForCausalLM": "pico_decoder.PicoDecoderHF"
+  },
+  "batch_size": 1024,
+  "d_model": 1536,
+  "max_seq_len": 2048,
+  "model_type": "pico_decoder",
+  "n_layers": 12,
+  "norm_eps": 1e-06,
+  "position_emb_theta": 10000.0,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
+  "vocab_size": 50304
+}

eval_results/step_0.json

@@ -0,0 +1 @@
+{"paloma": 60407.55679170296}

eval_results/step_1000.json

@@ -0,0 +1 @@
+{"paloma": 325.7754169842923}

eval_results/step_10000.json

@@ -0,0 +1 @@
+{"paloma": 39.63131210396929}

eval_results/step_100000.json

@@ -0,0 +1 @@
+{"paloma": 23.71192149534458}

eval_results/step_101000.json

@@ -0,0 +1 @@
+{"paloma": 23.722617558984393}

eval_results/step_102000.json

@@ -0,0 +1 @@
+{"paloma": 23.647181750673035}

eval_results/step_103000.json

@@ -0,0 +1 @@
+{"paloma": 23.617256277363474}

eval_results/step_104000.json

@@ -0,0 +1 @@
+{"paloma": 23.58370889288208}

eval_results/step_105000.json

@@ -0,0 +1 @@
+{"paloma": 23.574076971609003}

eval_results/step_106000.json

@@ -0,0 +1 @@
+{"paloma": 23.514877815196737}

eval_results/step_107000.json

@@ -0,0 +1 @@
+{"paloma": 23.512063555019658}

eval_results/step_108000.json

@@ -0,0 +1 @@
+{"paloma": 23.481759576348892}

eval_results/step_109000.json

@@ -0,0 +1 @@
+{"paloma": 23.42309071344781}

eval_results/step_11000.json

@@ -0,0 +1 @@
+{"paloma": 38.16542526977818}

eval_results/step_110000.json

@@ -0,0 +1 @@
+{"paloma": 23.396521913133018}

eval_results/step_111000.json

@@ -0,0 +1 @@
+{"paloma": 23.37710198400743}

eval_results/step_112000.json

@@ -0,0 +1 @@
+{"paloma": 23.309339042906146}

eval_results/step_113000.json

@@ -0,0 +1 @@
+{"paloma": 23.345229115552602}

eval_results/step_114000.json

@@ -0,0 +1 @@
+{"paloma": 23.29986716762237}

eval_results/step_115000.json

@@ -0,0 +1 @@
+{"paloma": 23.27356985670349}

eval_results/step_116000.json

@@ -0,0 +1 @@
+{"paloma": 23.228843119917016}

eval_results/step_117000.json

@@ -0,0 +1 @@
+{"paloma": 23.17763711808035}

eval_results/step_118000.json

@@ -0,0 +1 @@
+{"paloma": 23.129703529786564}

eval_results/step_119000.json

@@ -0,0 +1 @@
+{"paloma": 23.10688131553371}

eval_results/step_12000.json

@@ -0,0 +1 @@
+{"paloma": 37.00609017996838}

eval_results/step_120000.json

@@ -0,0 +1 @@
+{"paloma": 23.095857482255543}

eval_results/step_121000.json

@@ -0,0 +1 @@
+{"paloma": 23.07633466438134}

eval_results/step_122000.json

@@ -0,0 +1 @@
+{"paloma": 23.05234782687463}

eval_results/step_123000.json

@@ -0,0 +1 @@
+{"paloma": 22.985029866720325}

eval_results/step_124000.json

@@ -0,0 +1 @@
+{"paloma": 22.9857818419925}

eval_results/step_125000.json

@@ -0,0 +1 @@
+{"paloma": 22.96052921383223}

eval_results/step_13000.json

@@ -0,0 +1 @@
+{"paloma": 36.12258010915763}

eval_results/step_14000.json

@@ -0,0 +1 @@
+{"paloma": 35.30092575084873}

eval_results/step_15000.json

@@ -0,0 +1 @@
+{"paloma": 34.9831400311367}

eval_results/step_16000.json

@@ -0,0 +1 @@
+{"paloma": 33.944966741887534}

eval_results/step_17000.json

@@ -0,0 +1 @@
+{"paloma": 33.525249324692254}

eval_results/step_18000.json

@@ -0,0 +1 @@
+{"paloma": 33.06738855614479}

eval_results/step_19000.json

@@ -0,0 +1 @@
+{"paloma": 32.68154775822204}

eval_results/step_2000.json

@@ -0,0 +1 @@
+{"paloma": 893.744832137902}

eval_results/step_20000.json

@@ -0,0 +1 @@
+{"paloma": 32.365150609082875}

eval_results/step_21000.json

@@ -0,0 +1 @@
+{"paloma": 31.893736724238778}

eval_results/step_22000.json

@@ -0,0 +1 @@
+{"paloma": 31.542816529955182}

eval_results/step_23000.json

@@ -0,0 +1 @@
+{"paloma": 31.126356617498896}

eval_results/step_24000.json

@@ -0,0 +1 @@
+{"paloma": 30.837729862010438}

eval_results/step_25000.json

@@ -0,0 +1 @@
+{"paloma": 30.55359834328761}

eval_results/step_26000.json

@@ -0,0 +1 @@
+{"paloma": 30.223841104607132}

eval_results/step_27000.json

@@ -0,0 +1 @@
+{"paloma": 30.00440268965133}

eval_results/step_28000.json

@@ -0,0 +1 @@
+{"paloma": 29.728886223836227}