tokenizers.bpe
Byte Pair Encoding Text Tokenization
Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
0.1.4 |
2026-04-09 windows/windows R-4.5 | tokenizers.bpe_0.1.4.zip |
1.3 MiB |