kitoken
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Activity
- Latest release
- 3d ago
- Total releases
- 4
- Cadence
- ~6 months
- Last 12 months
- 2
Reach
- Downloads
- 48.6k
Details
- License
- BSD-2-Clause
- First release
- Dec 16, 2024
Releases