robotxt
The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
Activity
- Latest release
- 1w ago
- Total releases
- 15
- Cadence
- ~11 days
- Last 12 months
- 1
Reach
- Downloads
- 47.5k
Details
- License
- MIT
- First release
- Mar 11, 2023
Releases