guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
deep-learningrust-langtokenizertransformer
Created
Nov 2019
Total stars
345
Last 7 days
0
Last 30 days
0
Last 12 months
0
README badge
[](https://rustystars.dev/guillaume-be/rust-tokenizers)