discourse_ai-tokenizers
A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.
View on Github
Activity
- Latest release
- 6mo ago
- Total releases
- 10
- Cadence
- ~2 days
- Last 12 months
- 4
Reach
- Stars
- —
Details
- License
- MIT
- First release
- Jul 01, 2025
Releases
| Version | Released | |
|---|---|---|
0.4.2
patch
| ||
0.4.1
patch
| ||
0.4
minor
| ||
0.3.2
patch
| ||
0.3.1
patch
| ||
0.3.0
minor
| ||
0.2.0
minor
| ||
0.1.2
patch
| ||
0.1.1
patch
| ||
0.1.0
initial
|