tok

cranv0.2.2

Fast Text Tokenization. Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.

License MIT + file LICENSE0 versions1 maintainers2 deps3,452 weekly dl
mlverse/tok
58
/ 100
Health
safe to use

[email protected] is safe to use (health: 58/100)

Health breakdown0 – 100
25/25
maintenance
6/20
popularity
25/25
security
0/15
maturity
2/15
community
Vulnerabilities
0
none known

Health History

Dependency Tree

License Audit

Dependencies (2)
API access

Get this data programmatically — free, no authentication.

curl https://depscope.dev/api/check/cran/tok

First published · 2026-04-22 15:55:35

Last updated · 2026-04-22T13:40:02+00:00