GitHub topics

tokenizer

Repositories

16

Total stars

7,716

Trending

snchimata

Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.

Created Jul 2026

cli compression context-window cost-reduction developer-tools
Total352
30 days +234
lindera

A multilingual morphological analysis library.

Created Jan 2020

analyzer library morphological multilingual tokenizer
Total708
30 days +14
daac-tools

🎤 vibrato: Viterbi-based accelerated tokenizer

Created Jul 2022

japanese morphological-analysis nlp rust segmentation
Total430
30 days +2

Top

daac-tools

🎤 vibrato: Viterbi-based accelerated tokenizer

Created Jul 2022

japanese morphological-analysis nlp rust segmentation
Total430
snchimata

Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.

Created Jul 2026

cli compression context-window cost-reduction developer-tools
Total352
guillaume-be

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Created Nov 2019

deep-learning rust-lang tokenizer transformer
Total344
garvys-org

Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.

Created Sep 2018

asr automata composition finite-state-acceptors finite-state-transducers
Total191
nooscraft

CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.

Created Nov 2025

llms prompt-engineering rust tokenizer
Total146
Systemcluster

Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Created Jul 2023

bpe nlp nodejs python rust
Total64

Related

All topics →