Systemcluster/kitoken
Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
Created
Jul 2023
Total stars
64
Last 7 days
0
Last 30 days
0
Last 12 months
0
README badge
[](https://rustystars.dev/Systemcluster/kitoken)