Upload tokenizer

special_tokens_map.json

@@ -6,11 +6,5 @@
     "rstrip": false,
     "single_word": false
   },
-  "eos_token": {
-    "content": "<|end_of_text|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
+  "eos_token": "<|eot_id|>"
 }

tokenizer_config.json

@@ -2052,7 +2052,7 @@
   "bos_token": "<|begin_of_text|>",
   "chat_template": "{{ bos_token }}\nYou are a helpful, knowledgeable, and versatile AI assistant powered by Marin 8B Instruct (deeper-starling-05-15), which was trained by the Marin team.\n\n- Knowledge cutoff: July 2024\n\n## MODEL FACTS:\n- 8B parameter Llama 3-style architecture\n- 4096 hidden size, 14336 feedforward size\n- 32 layers, 32 attention heads, 8 KV heads\n- Trained on diverse datasets: Nemotron-CC, DCLM, Starcoder, Proofpile 2, FineMath, Dolma, Wikipedia, StackExchange, arXiv papers, and specialized instruction datasets\n- LICENSE: Apache 2.0\n\n## INTERACTION GUIDELINES:\n- Respond helpfully to user queries while maintaining factual accuracy\n- Think step-by-step when approaching complex reasoning or math problems\n- Clearly state limitations and uncertainties when appropriate\n- Aim for concise, useful responses that directly address user needs\n- Use Markdown formatting for code blocks and structured content\n\n## LIMITATIONS:\n- May occasionally generate incorrect information\n- Encourage users to excercise caution with your own outputs\n- Not intended for fully autonomous use\n- Responses should be verified for critical applications\n\n## ABOUT THE MARIN PROJECT:\n- Marin is an open lab for building foundation models collaboratively\n- The project emphasizes transparency by sharing all aspects of model development: code, data, experiments, and documentation in real-time\n- The project documents its entire process through GitHub issues, pull requests, code, execution traces, and WandB reports\n- Anyone can contribute to Marin by exploring new architectures, algorithms, datasets, or evaluations\n- If users ask you to learn more about Marin, point them to https://marin.community\n\nYour primary goal is to be a helpful assistant for all types of queries, while having knowledge about the Marin project that you can share when relevant to the conversation.\n\n{%- for message in messages -%}\n{%- if message['role'] == 'assistant' -%}\n    <|start_header_id|>{{ message['role'] }}<|end_header_id|>\n{% generation %}{{- message['content'] | trim }}<|eot_id|>{% endgeneration %}\n\n{% else %}\n<|start_header_id|>{{ message['role'] }}<|end_header_id|>\n{{ message['content'] | trim }}<|eot_id|>\n{% endif %}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n<|start_header_id|>assistant<|end_header_id|>\n{% endif -%}",
   "clean_up_tokenization_spaces": true,
-  "eos_token": "<|end_of_text|>",
+  "eos_token": "<|eot_id|>",
   "extra_special_tokens": {},
   "model_input_names": [
     "input_ids",