spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules
The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
Versions across snapshots
| Version | Repository | File | Size |
|---|---|---|---|
0.2.5 |
rolling linux/jammy R-4.5 | spiderbar_0.2.5.tar.gz |
118.1 KiB |
0.2.5 |
rolling linux/noble R-4.5 | spiderbar_0.2.5.tar.gz |
128.4 KiB |
0.2.5 |
rolling source/ R- | spiderbar_0.2.5.tar.gz |
31.0 KiB |
0.2.5 |
latest linux/jammy R-4.5 | spiderbar_0.2.5.tar.gz |
118.1 KiB |
0.2.5 |
latest linux/noble R-4.5 | spiderbar_0.2.5.tar.gz |
128.4 KiB |
0.2.5 |
latest source/ R- | spiderbar_0.2.5.tar.gz |
31.0 KiB |
0.2.5 |
2026-04-26 source/ R- | spiderbar_0.2.5.tar.gz |
31.0 KiB |
0.2.5 |
2026-04-23 source/ R- | spiderbar_0.2.5.tar.gz |
31.0 KiB |
0.2.5 |
2026-04-09 windows/windows R-4.5 | spiderbar_0.2.5.zip |
459.3 KiB |
0.2.5 |
2025-04-20 source/ R- | spiderbar_0.2.5.tar.gz |
31.0 KiB |