rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
0.0.7 |
rolling linux/jammy R-4.5 | rtiktoken_0.0.7.tar.gz |
4.5 MiB |
0.0.7 |
rolling linux/noble R-4.5 | rtiktoken_0.0.7.tar.gz |
4.5 MiB |
0.0.7 |
rolling source/ R- | rtiktoken_0.0.7.tar.gz |
7.9 MiB |
0.0.7 |
latest linux/jammy R-4.5 | rtiktoken_0.0.7.tar.gz |
4.5 MiB |
0.0.7 |
latest linux/noble R-4.5 | rtiktoken_0.0.7.tar.gz |
4.5 MiB |
0.0.7 |
latest source/ R- | rtiktoken_0.0.7.tar.gz |
7.9 MiB |
0.0.7 |
2026-04-26 source/ R- | rtiktoken_0.0.7.tar.gz |
7.9 MiB |
0.0.7 |
2026-04-23 source/ R- | rtiktoken_0.0.7.tar.gz |
7.9 MiB |
0.0.7 |
2026-04-09 windows/windows R-4.5 | rtiktoken_0.0.7.zip |
4.3 MiB |
0.0.7 |
2025-04-20 source/ R- | rtiktoken_0.0.7.tar.gz |
7.9 MiB |
Dependencies (latest)
Suggests
- testthat (>= 3.0.0)