3 ms·
What are you trying to scrape? This is HN's robots.txt: User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disal
by _delirium 17y ago
What are you trying to scrape? This is HN's robots.txt:
User-Agent: *
Disallow: /x?
Disallow: /vote?
Disallow: /reply?
Disallow: /submitted?
Disallow: /threads?
That doesn't look like it blocks either the front page, the /newest page, or article pages (which have the form /item?id=N).
- gmaster1440 17y agotry: http://developer.yahoo.com/yql/console/?q=select%20*%20from%20html%20where%20url%3D%22http://news.ycombinator.com%22%20and%20xpath%3D//table//table//td[%40class%3D%22title%22]/a#h=select%20*%20from%20html%20where%20url%3D%22http%3A//news.ycombinator.com%22%20and%20xpath%3D%27//table//table//td%5B@class%3D%22title%22%5D/a%27 http://developer.yahoo.com/yql/console/?q=select%20*%20from%...
- _delirium 17y agoWorks for me--- I clicked "test" and got what looks like a list of HN's front-page links in the results.
- gmaster1440 17y agovery weird, this is what i get: http://img444.imageshack.us/img444/290/screenshot20100413at813.png http://img444.imageshack.us/img444/290/screenshot20100413at8...