Lukas Helff commited on
Commit
1fe4885
·
1 Parent(s): ac97ee4

make eval config not obligatory

Browse files
VerifiableRewardsForScalableLogicalReasoning.py CHANGED
@@ -91,7 +91,7 @@ Args:
91
  references (`list` of `dict`): Each reference should contain:
92
  - 'validation_program' (`str`): Background knowledge in Prolog syntax
93
  - 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
94
- Define: positive_predicate, and negative_predicate
95
  Returns:
96
  accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
97
  partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
@@ -261,10 +261,7 @@ class VerifiableRewardsForScalableLogicalReasoning(evaluate.Metric):
261
  'predictions': datasets.Value('string'),
262
  'references': {
263
  'validation_program': datasets.Value('string'),
264
- 'evaluation_config': {
265
- 'positive_predicate': datasets.Value('string'),
266
- 'negative_predicate': datasets.Value('string')
267
- }
268
  },
269
  }),
270
  codebase_urls=["https://github.com/AIML-TUDA/SLR-Bench"],
 
91
  references (`list` of `dict`): Each reference should contain:
92
  - 'validation_program' (`str`): Background knowledge in Prolog syntax
93
  - 'evaluation_config' (`dict`, optional): Configuration of predicates to use for evaluation.
94
+ Define: positive_predicate, and negative_predicate, the positive one should match the head of the rule to evaluate.
95
  Returns:
96
  accuracy (`float`): The proportion of predictions that correctly solve all examples. Value is between 0 and 1.
97
  partial_score (`float`): Average proportion of correctly classified examples across all predictions. Value is between 0 and 1.
 
261
  'predictions': datasets.Value('string'),
262
  'references': {
263
  'validation_program': datasets.Value('string'),
264
+ 'evaluation_config': datasets.Value("dict", id=None)
 
 
 
265
  },
266
  }),
267
  codebase_urls=["https://github.com/AIML-TUDA/SLR-Bench"],