Sign in Sign up
rubygems

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

rubygems View on Rubygems

Activity

Latest release
5y ago
Total releases
94
Cadence
~5 days
Last 12 months
0

Details

License
MIT
First release
Nov 10, 2010
Releases
Version Released
1.2.1 patch
1.2.0 minor
1.1.0 minor
1.0.29 patch
1.0.28 patch
1.0.27 patch
1.0.26 patch
1.0.25 patch
1.0.24 patch
1.0.22 patch
1.0.23 patch
1.0.21 patch
1.0.20 patch
1.0.19 patch
1.0.18 patch
1.0.17 patch
1.0.16 patch
1.0.15 patch
1.0.12 patch
1.0.11 patch
1.0.9 patch
1.0.10 patch
1.0.8 patch
1.0.6 patch
1.0.5 patch
1.0.4 patch
1.0.3 patch
1.0.2 patch
1.0.1 patch
1.0.0 major
0.0.77 patch
0.0.76 patch
0.0.75 patch
0.0.74 patch
0.0.73 patch
0.0.72 patch
0.0.70 patch
0.0.71 patch
0.0.68 patch
0.0.67 patch
0.0.66 patch
0.0.65 patch
0.0.64 patch
0.0.63 patch
0.0.62 patch
0.0.61 patch
0.0.60 patch
0.0.59 patch
0.0.58 patch
0.0.57 patch
1–50 of 94