💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Last push 2 hours ago

nlpnatural-language-processingnatural-language-understandinglanguage-modeltransformersbertgpt

Created

Nov 2019

Total stars

11,258

Last 7 days

0

Last 30 days

0

Last 12 months

0

README badge

Rusty Stars badge for huggingface/tokenizers

Show this repo's rank among all Rust repos. Paste the Markdown into the README.

[![Rusty Stars](https://rustystars.dev/badge/huggingface/tokenizers.svg)](https://rustystars.dev/huggingface/tokenizers)