llm-tokenizer
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
Activity
- Latest release
- 3y ago
- Total releases
- 2
- Cadence
- ~5 days
- Last 12 months
- 0
Details
- License
- Apache-2.0
- First release
- Sep 02, 2023
Releases
| Version | Released | |
|---|---|---|
1.0.2
patch
| ||
1.0.1
initial
|