Crandore Hub

piecemaker

Tools for Preparing Text for Tokenizers

Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.

Versions across snapshots

VersionRepositoryFileSize
1.0.2 rolling linux/jammy R-4.5 piecemaker_1.0.2.tar.gz 39.3 KiB
1.0.2 rolling linux/noble R-4.5 piecemaker_1.0.2.tar.gz 39.1 KiB
1.0.2 rolling source/ R- piecemaker_1.0.2.tar.gz 12.3 KiB
1.0.2 latest linux/jammy R-4.5 piecemaker_1.0.2.tar.gz 39.3 KiB
1.0.2 latest linux/noble R-4.5 piecemaker_1.0.2.tar.gz 39.1 KiB
1.0.2 latest source/ R- piecemaker_1.0.2.tar.gz 12.3 KiB
1.0.2 2026-04-26 source/ R- piecemaker_1.0.2.tar.gz 12.3 KiB
1.0.2 2026-04-23 source/ R- piecemaker_1.0.2.tar.gz 12.3 KiB
1.0.2 2026-04-09 windows/windows R-4.5 piecemaker_1.0.2.zip 42.7 KiB
1.0.2 2025-04-20 source/ R- piecemaker_1.0.2.tar.gz 12.3 KiB

Dependencies (latest)

Imports

Suggests