Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Last push 10 months ago
Stars per month
month Stars
2020-01 5
2020-02 2
2020-03 6
2020-04 6
2020-05 6
2020-06 8
2020-07 6
2020-08 3
2020-09 4
2020-10 6
2020-11 8
2020-12 4
2021-01 5
2021-02 1
2021-03 3
2021-04 4
2021-05 4
2021-06 3
2021-07 4
2021-08 5
2021-09 3
2021-10 2
2021-11 3
2021-12 3
2022-01 3
2022-02 4
2022-03 1
2022-04 2
2022-05 2
2022-06 2
2022-07 2
2022-08 2
2022-09 4
2022-10 6
2022-11 5
2022-12 21
2023-01 7
2023-02 10
2023-03 5
2023-04 15
2023-05 8
2023-06 6
2023-07 6
2023-08 4
2023-09 6
2023-10 4
2023-11 6
2023-12 3
2024-01 8
2024-02 8
2024-03 7
2024-04 5
2024-05 2
2024-06 3
2024-07 3
2024-08 5
2024-09 6
2024-10 3
2024-11 2
2024-12 2
2025-01 1
2025-02 2
2025-03 2
2025-04 9
2025-05 4
2025-06 4
2025-07 3
2025-08 2
2025-09 2
2025-10 2
2025-11 1
2025-12 2
2026-01 3
2026-02 2
2026-03 3
2026-04 2
2026-05 3
2026-06 2
2026-08 1
2026-09 1
2026-10 1
← 1y
1y →
README badge
[](https://rustystars.dev/guillaume-be/rust-tokenizers)
Copy