marcelroed

marcelroed/gigatoken

Language model tokenization at GB/s

Last push 2 weeks ago

llmnlptokenizationtokenizer

Created

Nov 2025

Total stars

4,157

Last 7 days

+3

Last 30 days

+29

Last 12 months

+4,157

README badge

Rusty Stars badge for marcelroed/gigatoken
[![Rusty Stars](https://rustystars.dev/badge/marcelroed/gigatoken.svg)](https://rustystars.dev/marcelroed/gigatoken)