Tiktoken.Encodings.Tokenizer
Load tokenizer.json (HuggingFace format) as a Tiktoken encoding.
Activity
- Latest release
- 3d ago
- Total releases
- 20
- Cadence
- ~daily
- Last 12 months
- 20
Details
- License
- MIT
- First release
- Mar 18, 2026
| Version | Released | |
|---|---|---|
3.1.6-alpha.0.22
pre
|
3.1.6-alpha.0.22
pre
Dependencies (3)
|
|
3.1.6-alpha.0.21
pre
|
3.1.6-alpha.0.21
pre
Dependencies (3)
|
|
3.1.6-alpha.0.20
pre
|
3.1.6-alpha.0.20
pre
Dependencies (3)
|
|
3.1.5
patch
|
3.1.5
patch
Dependencies (3)
|
|
3.1.5-alpha.0.6
pre
|
3.1.5-alpha.0.6
pre
Dependencies (3)
|
|
3.1.4
patch
|
3.1.4
patch
Dependencies (3)
|
|
3.1.3
patch
|
3.1.3
patch
Dependencies (3)
|
|
3.1.2
patch
|
3.1.2
patch
Dependencies (3)
|
|
3.1.2-alpha.0.1
pre
|
3.1.2-alpha.0.1
pre
Dependencies (3)
|
|
3.1.1
patch
|
3.1.1
patch
Dependencies (3)
|
|
3.1.1-alpha.0.1
pre
|
3.1.1-alpha.0.1
pre
Dependencies (3)
|
|
3.1.0
minor
|
3.1.0
minor
Dependencies (3)
|
|
3.1.0-rc.1.4
pre
|
3.1.0-rc.1.4
pre
Dependencies (3)
|
|
3.1.0-rc.1.3
pre
|
3.1.0-rc.1.3
pre
Dependencies (3)
|
|
3.0.1-alpha.0.4
pre
|
3.0.1-alpha.0.4
pre
Dependencies (3)
|
|
3.1.0-rc.1.2
pre
|
3.1.0-rc.1.2
pre
Dependencies (3)
|
|
3.1.0-rc.1.1
pre
|
3.1.0-rc.1.1
pre
Dependencies (3)
|
|
3.1.0-rc.1
pre
|
3.1.0-rc.1
pre
Dependencies (3)
|
|
3.0.0
major
|
3.0.0
major
Dependencies (3)
|
|
2.2.0
initial
|
2.2.0
initial
Dependencies (3)
|