Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Last push 3 weeks ago

bpenlpnodejspythonrustsentencepiecetokenizerunigramwebword-segmentation

Created

Jul 2023

Total stars

64

Last 7 days

0

Last 30 days

0

Last 12 months

0

README badge

Rusty Stars badge for Systemcluster/kitoken
[![Rusty Stars](https://rustystars.dev/badge/Systemcluster/kitoken.svg)](https://rustystars.dev/Systemcluster/kitoken)