Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.
Created Jul 2026
Repositories
16
Total stars
7,716
Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.
Created Jul 2026
Language model tokenization at GB/s
Created Nov 2025
Ready-made tokenizer library for working with GPT and tiktoken
Created Feb 2023
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
Created Aug 2021
🎤 vibrato: Viterbi-based accelerated tokenizer
Created Jul 2022
Language model tokenization at GB/s
Created Nov 2025
🎤 vibrato: Viterbi-based accelerated tokenizer
Created Jul 2022
Ready-made tokenizer library for working with GPT and tiktoken
Created Feb 2023
Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.
Created Jul 2026
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Created Nov 2019
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
Created Aug 2021
Rust re-implementation of OpenFST - library for constructing, combining, optimizing, and searching weighted finite-state transducers (FSTs). A Python binding is also available.
Created Sep 2018
Chinese tokenizer for tantivy, based on jieba-rs
Created Sep 2018
Fast tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
Created Jul 2023
A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.
Created Nov 2025