SentencePieceTokenizer
SentencePieceTokenizer is a wrapper around the google SentencePiece tokenizer. Used to tokenize text for language models and other NLP tasks.
View on Github
Activity
- Latest release
- 6mo ago
- Total releases
- 6
- Cadence
- ~7 days
- Last 12 months
- 2
Reach
- Stars
- —
Details
- First release
- Dec 29, 2024
Releases
| Version | Released | |
|---|---|---|
0.1.6
patch
| ||
0.1.5
patch
| ||
0.1.4
patch
| ||
0.1.3
patch
| ||
0.1.2
patch
| ||
0.1.1
initial
|