Sign in Sign up

npm

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Activity

Latest release: 1w ago
Total releases: 39
Cadence: ~5 days
Last 12 months: 1

Reach

Stars: 11.0k

Details

License: Apache-2.0
First release: May 15, 2019

Releases

Version	Released
`0.23.2` minor	Sep 03, 2026
`0.13.4-rc2` pre	May 17, 2023
`0.13.4-rc1` pre	May 17, 2023
`0.13.3` patch	May 14, 2023
`0.13.1` patch	Oct 06, 2022
`0.13.0` minor	Sep 19, 2022
`0.12.0` minor deprecated	Mar 31, 2022
`0.8.2` patch	Jan 17, 2022
`0.8.1` patch	Jan 17, 2022
`0.8.0` minor	Sep 02, 2021
`0.7.0` minor	Jul 01, 2020
`0.6.2` patch	Apr 13, 2020
`0.6.1` patch	Apr 01, 2020
`0.6.0` minor	Mar 30, 2020
`0.5.0` minor	Feb 27, 2020
`0.4.1` patch	Feb 11, 2020
`0.4.0` minor	Feb 05, 2020
`0.3.1` patch	Jan 24, 2020
`0.3.0` minor	Jan 23, 2020
`0.2.4` patch	Jan 14, 2020
`0.2.3` patch	Jan 14, 2020
`0.2.2` patch	Jan 10, 2020
`0.2.1` patch	Jan 10, 2020
`0.2.0` minor	Jan 10, 2020
`0.1.14` patch	Nov 27, 2019
`0.1.13` patch	Jul 19, 2019
`0.1.12` patch	Jul 16, 2019
`0.1.11` patch	Jul 15, 2019
`0.1.10` patch	Jun 23, 2019
`0.1.9` patch	Jun 19, 2019
`0.1.8` patch	Jun 08, 2019
`0.1.7` patch	Jun 01, 2019
`0.1.6` patch	May 28, 2019
`0.1.5` patch	May 26, 2019
`0.1.4` patch	May 25, 2019
`0.1.3` patch	May 23, 2019
`0.1.2` patch	May 23, 2019
`0.1.1` patch	May 20, 2019
`0.1.0` initial	May 15, 2019

Repository

huggingface/tokenizers

Forks: 1.2k
Open issues: 119
Language: Rust
: 5.8/ 10

nlp
natural-language-processing
natural-language-understanding
language-model
transformers
bert

Dependencies

Used by

14

Supply-chain exposure

14 tracked projects depend on this, <1% of every package we track.

Insights

Releases per month

last 12 months

Release mix

minor 10
patch 26
pre 2

39 releases

Dependencies over time

per release

Minimum runtime

over time

Release calendar

2026

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Less More