guillaume-be

guillaume-be/rust-tokenizers

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Last push 10 months ago

deep-learningrust-langtokenizertransformer

Created

Nov 2019

Total stars

345

Last 7 days

0

Last 30 days

0

Last 12 months

0

README badge

Rusty Stars badge for guillaume-be/rust-tokenizers
[![Rusty Stars](https://rustystars.dev/badge/guillaume-be/rust-tokenizers.svg)](https://rustystars.dev/guillaume-be/rust-tokenizers)