AtsuMiyai commited on
Commit
d5ca00f
·
1 Parent(s): 66d5f52

update constants.py

Browse files
Files changed (1) hide show
  1. constants.py +21 -20
constants.py CHANGED
@@ -32,32 +32,33 @@ JMMMU consists of **720 translation-based (Culture Agnostic)** and **600 brand-n
32
  """
33
 
34
 
35
- SUBMIT_INTRODUCTION = """# Submit on JMMMU Benchmark Introduction
36
- We do not recommend including results obtained from extensive prompt engineering since it is important to prevent performance hacking and better reflect real-world use cases.
 
37
  For more details, please refer to the [lmms-eval code base](https://github.com/EvolvingLMMs-Lab/lmms-eval) and the upcoming paper (coming soon).
38
 
39
- 1. Obtain Result JSON File from [lmms-eval code base](https://github.com/EvolvingLMMs-Lab/lmms-eval).
40
- 2. If you want to update existing model performance by uploading new results, please ensure 'Model Name Revision' is the same as what's shown in the leaderboard. For example, if you want to modify LLaVA-OV 7B's performance, you need to fill in 'LLaVA-OV 7B' in 'Revision Model Name'.
41
- 3. Please provide the correct link of your model's repository for each submission.
42
- 4. After clicking 'Submit Eval', you can click 'Refresh' to obtain the latest result in the leaderboard.
43
 
44
- Note: The example of the submitted JSON file is this url: [result.json](https://drive.google.com/file/d/10CF1c24BhoK9OM8De-2gLXDDnNcnMWvy/view?usp=sharing).
45
 
46
- ## Submit Example
47
- If you want to upload LLaVA-OV 7B's result in the leaderboard, you need to:
48
- 1. Select LMM in 'Model Type'.
49
- 2. Fill in 'LLaVA-OV 7B' in 'Model Name' if it is your first time to submit your result (You can leave 'Revision Model Name' blank).
50
- 3. Fill in 'LLaVA-OV 7B' in 'Revision Model Name' if you want to update your result (You can leave 'Model Name' blank).
51
- 4. Fill in 'https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov' in 'Model Link'.
52
- 5. Fill in '7B' in 'Model size'.
53
- 6. Upload results.json.
54
- 7. Click the 'Submit Eval' button.
55
- 8. Click 'Refresh' to obtain the uploaded leaderboard.
56
 
57
- To check whether the submission is successful, you can click the 'Logs' button. If the message 'Success! Your submission has been added!' appears, the submission is successful.
58
 
59
- ### If you have any questions or deletion requests, please contact [miyai@cvm.t.u-tokyo.ac.jp](miyai@cvm.t.u-tokyo.ac.jp).
60
- ### ⚠️ Please do not submit any malicious file (e.g, files you manually edited).
61
  """
62
 
63
 
 
32
  """
33
 
34
 
35
+ SUBMIT_INTRODUCTION = """# Submit on JMMMU Benchmark
36
+ ## Introduction
37
+ We do not recommend including results obtained from extensive prompt engineering since it is important to prevent performance hacking and better reflect real-world use cases.
38
  For more details, please refer to the [lmms-eval code base](https://github.com/EvolvingLMMs-Lab/lmms-eval) and the upcoming paper (coming soon).
39
 
40
+ 1. Obtain Result JSON File from [lmms-eval code base](https://github.com/EvolvingLMMs-Lab/lmms-eval).
41
+ 2. If you want to update existing model performance by uploading new results, please ensure 'Model Name Revision' is the same as what's shown in the leaderboard. For example, if you want to modify LLaVA-OV 7B's performance, you need to fill in 'LLaVA-OV 7B' in 'Revision Model Name'.
42
+ 3. Please provide the correct link of your model's repository for each submission.
43
+ 4. After clicking 'Submit Eval', you can click 'Refresh' to obtain the latest result in the leaderboard.
44
 
45
+ Note: The example of the submitted JSON file is this url: [result.json](https://drive.google.com/file/d/10CF1c24BhoK9OM8De-2gLXDDnNcnMWvy/view?usp=sharing).
46
 
47
+ ## Submit Example
48
+ If you want to upload LLaVA-OV 7B's result in the leaderboard, you need to:
49
+ 1. Select LMM in 'Model Type'.
50
+ 2. Fill in 'LLaVA-OV 7B' in 'Model Name' if it is your first time to submit your result (You can leave 'Revision Model Name' blank).
51
+ 3. Fill in 'LLaVA-OV 7B' in 'Revision Model Name' if you want to update your result (You can leave 'Model Name' blank).
52
+ 4. Fill in 'https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov' in 'Model Link'.
53
+ 5. Fill in '7B' in 'Model size'.
54
+ 6. Upload results.json.
55
+ 7. Click the 'Submit Eval' button.
56
+ 8. Click 'Refresh' to obtain the uploaded leaderboard.
57
 
58
+ To check whether the submission is successful, you can click the 'Logs' button. If the message 'Success! Your submission has been added!' appears, the submission is successful.
59
 
60
+ ### If you have any questions or deletion requests, please contact [miyai@cvm.t.u-tokyo.ac.jp](miyai@cvm.t.u-tokyo.ac.jp).
61
+ ### ⚠️ Please do not submit any malicious file (e.g, files you manually edited).
62
  """
63
 
64