tokenizers
Fast, Consistent Tokenization of Natural Language Text
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
0.3.0 |
rolling linux/jammy R-4.5 | tokenizers_0.3.0.tar.gz |
626.4 KiB |
0.3.0 |
rolling linux/noble R-4.5 | tokenizers_0.3.0.tar.gz |
628.6 KiB |
0.3.0 |
rolling source/ R- | tokenizers_0.3.0.tar.gz |
447.0 KiB |
0.3.0 |
latest linux/jammy R-4.5 | tokenizers_0.3.0.tar.gz |
626.4 KiB |
0.3.0 |
latest linux/noble R-4.5 | tokenizers_0.3.0.tar.gz |
628.6 KiB |
0.3.0 |
latest source/ R- | tokenizers_0.3.0.tar.gz |
447.0 KiB |
0.3.0 |
2026-04-26 source/ R- | tokenizers_0.3.0.tar.gz |
447.0 KiB |
0.3.0 |
2026-04-23 source/ R- | tokenizers_0.3.0.tar.gz |
447.0 KiB |
0.3.0 |
2026-04-09 windows/windows R-4.5 | tokenizers_0.3.0.zip |
954.2 KiB |
0.3.0 |
2025-04-20 source/ R- | tokenizers_0.3.0.tar.gz |
447.0 KiB |