webcrawler « nutch « Java Lucene Q&A

1. Why doesn't Nutch seem to know about "Last-Modified"?    stackoverflow.com

I setup Nutch with a db.fetch.interval.default of 60000 so that I can crawl every day. If I don't, it won't even look at my site when I crawl the next ...

2. Does any open, simply extendible web crawler exists?    stackoverflow.com

I search for a web crawler solution which can is mature enough and can be simply extended. I am interested in the following features... or possibility to extend the crawler to ...

3. Nutch - how to crawl by small patches?    stackoverflow.com

I am stuck! Can`t get Nutch to crawl for me by small patches. I start it by bin/nutch crawl command with parameters -depth 7 and -topN 10000. And it never ends. ...

4. Getting nutch to prioritize frequently updated pages?    stackoverflow.com

Is there a way to get Nutch to increase the crawling of pages that gets updated frequently? E.g. index pages and feeds. It would also be of value to refresh fresh pages ...

5. Re-crawling websites fast    stackoverflow.com

I am developing a system that has to track content of few portals and check changes every night (for example download and index new sites that have been added during the ...

6. how to crawl a specific URL using nutch 1.2    stackoverflow.com

I'm using nutch-1.2 but not able to restrict my config file to crawl only given urls
my crawl-urlfilter.txt file is

    # Each non-comment, non-blank line contains a regular expression
 ...