I am trying to crawl a page that requires Siteminder Authentication, So I am trying to pass my username and password in the code itself to access that page and keep ...
Hey, I need to write a http data source that crawl a given site , send the HTML content to a service and store all the url that accessible in some Data Structure(Can be a DB). I want that this crawler will support in cookies and it will be great if it will support in "hot update". Can any one recommend ...