robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
0.7.15 |
rolling linux/jammy R-4.5 | robotstxt_0.7.15.tar.gz |
195.7 KiB |
0.7.15 |
rolling linux/noble R-4.5 | robotstxt_0.7.15.tar.gz |
195.2 KiB |
0.7.15 |
rolling source/ R- | robotstxt_0.7.15.tar.gz |
132.9 KiB |
0.7.15 |
latest linux/jammy R-4.5 | robotstxt_0.7.15.tar.gz |
195.7 KiB |
0.7.15 |
latest linux/noble R-4.5 | robotstxt_0.7.15.tar.gz |
195.2 KiB |
0.7.15 |
latest source/ R- | robotstxt_0.7.15.tar.gz |
132.9 KiB |
0.7.15 |
2026-04-26 source/ R- | robotstxt_0.7.15.tar.gz |
132.9 KiB |
0.7.15 |
2026-04-23 source/ R- | robotstxt_0.7.15.tar.gz |
132.9 KiB |
0.7.15 |
2026-04-09 windows/windows R-4.5 | robotstxt_0.7.15.zip |
208.7 KiB |
0.7.15 |
2025-04-20 source/ R- | robotstxt_0.7.15.tar.gz |
132.9 KiB |