Spaces:

ZurichNLP
/

subword-tokenization

Running

jvamvas commited on 29 days ago

Commit

1390fb6

1 Parent(s): a35d485

Use gsw vocab from hf hub

Files changed (1) hide show

utils.py CHANGED Viewed

@@ -70,8 +70,7 @@ def load_gsw_tokenizer() -> Tuple[str, object]:
     Returns:
         Tuple of (tokenizer_name, tokenizer_object)
     """
-    tokenizer_path = Path(__file__).parent / "gsw_tokenizer"
-    tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
     return "swissbert-gsw", tokenizer
 def load_tokenizers() -> Dict[str, object]:

     Returns:
         Tuple of (tokenizer_name, tokenizer_object)
     """
+    tokenizer = AutoTokenizer.from_pretrained("jvamvas/swissbert-gsw-vocab")
     return "swissbert-gsw", tokenizer
 def load_tokenizers() -> Dict[str, object]: