3 ms·
very cool, is it doing a depth first blind crawl of any domain you throw at it?
by volokoumphetico 13y ago
very cool, is it doing a depth first blind crawl of any domain you throw at it?
- calufa 13y agoIt will basically go through all the links it finds, that can be millions of links. You can also tell it to ignore certain links using regular expressions or via tales java apis.
- calufa 13y agoHere is a sample code of a 1 depth scrape: https://github.com/calufa/tales-templates/blob/master/core/src/sample/FirstDepthTemplate.java https://github.com/calufa/tales-templates/blob/master/core/s... This is the api call to start the scraper on twitter: http://localhost:8080/start?process=tales.scrapers.LoopScraper http://localhost:8080/start?process=tales.scrapers.LoopScrap... -template tales.templates.FirstDepthTemplate -threads 2 -namespace com_twitter -baseURL twitter.com