guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Created
Nov 2019
Total stars
344
Last 7 days
0
Last 30 days
0
Last 12 months
+21
README badge
[](https://rustystars.dev/guillaume-be/rust-tokenizers)