morphemepiece
Morpheme Tokenization
Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
1.2.3 |
rolling linux/jammy R-4.5 | morphemepiece_1.2.3.tar.gz |
72.0 KiB |
1.2.3 |
rolling linux/noble R-4.5 | morphemepiece_1.2.3.tar.gz |
71.9 KiB |
1.2.3 |
rolling source/ R- | morphemepiece_1.2.3.tar.gz |
55.6 KiB |
1.2.3 |
latest linux/jammy R-4.5 | morphemepiece_1.2.3.tar.gz |
72.0 KiB |
1.2.3 |
latest linux/noble R-4.5 | morphemepiece_1.2.3.tar.gz |
71.9 KiB |
1.2.3 |
latest source/ R- | morphemepiece_1.2.3.tar.gz |
55.6 KiB |
1.2.3 |
2026-04-26 source/ R- | morphemepiece_1.2.3.tar.gz |
55.6 KiB |
1.2.3 |
2026-04-23 source/ R- | morphemepiece_1.2.3.tar.gz |
55.6 KiB |
1.2.3 |
2026-04-09 windows/windows R-4.5 | morphemepiece_1.2.3.zip |
80.7 KiB |
1.2.3 |
2025-04-20 source/ R- | morphemepiece_1.2.3.tar.gz |
55.6 KiB |