crawlberg-robots
The robots.txt parser shared by the crawlberg crawler and its headless-browser fallback.
Activity
- Latest release
- 2d ago
- Total releases
- 10
- Cadence
- ~daily
- Last 12 months
- 10
Reach
- Downloads
- 6.6k
Details
- License
- MIT
- First release
- Oct 03, 2026
Releases
| Version | Released | |
|---|---|---|
1.12.3
patch
| ||
1.12.2
patch
| ||
1.12.0
minor
| ||
1.11.1
patch
| ||
1.11.0
minor
| ||
1.10.3
patch
| ||
1.10.2
patch
| ||
1.10.1
patch
| ||
1.10.0
minor
| ||
1.9.0
initial
|