Lukas Helff
commited on
Commit
·
1fe4885
1
Parent(s):
ac97ee4
make eval config not obligatory
Browse files
VerifiableRewardsForScalableLogicalReasoning.py
CHANGED
@@ -91,7 +91,7 @@ Args:
|
|
91 |
references (`list` of `dict`): Each reference should contain:
|
92 |
- 'validation_program' (`str`): Background knowledge in Prolog syntax
|
93 |
- 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
|
94 |
-
Define: positive_predicate, and negative_predicate
|
95 |
Returns:
|
96 |
accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
|
97 |
partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
|
@@ -261,10 +261,7 @@ class VerifiableRewardsForScalableLogicalReasoning(evaluate.Metric):
|
|
261 |
'predictions': datasets.Value('string'),
|
262 |
'references': {
|
263 |
'validation_program': datasets.Value('string'),
|
264 |
-
'evaluation_config':
|
265 |
-
'positive_predicate': datasets.Value('string'),
|
266 |
-
'negative_predicate': datasets.Value('string')
|
267 |
-
}
|
268 |
},
|
269 |
}),
|
270 |
codebase_urls=["https://github.com/AIML-TUDA/SLR-Bench"],
|
|
|
91 |
references (`list` of `dict`): Each reference should contain:
|
92 |
- 'validation_program' (`str`): Background knowledge in Prolog syntax
|
93 |
- 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
|
94 |
+
Define: positive_predicate, and negative_predicate, the positive one should match the head of the rule to evaluate.
|
95 |
Returns:
|
96 |
accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
|
97 |
partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
|
|
|
261 |
'predictions': datasets.Value('string'),
|
262 |
'references': {
|
263 |
'validation_program': datasets.Value('string'),
|
264 |
+
'evaluation_config': datasets.Value("dict", id=None)
|
|
|
|
|
|
|
265 |
},
|
266 |
}),
|
267 |
codebase_urls=["https://github.com/AIML-TUDA/SLR-Bench"],
|