language-tokenizer
Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.
View on Github
Activity
- Latest release
- 3w ago
- Total releases
- 5
- Cadence
- ~2 months
- Last 12 months
- 5
Reach
- Downloads
- 58.2k
- Stars
- 2
Details
- License
- unknown
- First release
- Jan 07, 2026
Releases