3 ms·
> AngelList even detects PhamtomJS (have not seen other sites do this). I run a site that aggregates/crawls job boards for remote job postings, and AngelList
by mosburger 10y ago
> AngelList even detects PhamtomJS (have not seen other sites do this).
I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and will likely retire that bot soon.
It kinda sucks that I'm just grabbing publicly-available content in a very low-bandwidth way, but I really can't convince myself that what I'm doing is very ethical.
My to-do list includes making my crawler into a more well-behaved bot and that will have to go.
- franciskim 10y agoI think PhantomJS has a bit of a giveaway in the headers where 2 lines are reversed compared to a normal Chrome, I always thought AngelList detected this flaw. Although I have heard there are builds of PhantomJS where this flaw does not exist.
- ixtli 10y agoI think you may want to decouple your ethical analysis from which private company is making the most money. Remember that the only functional difference between you and somewhere like kayak.com or padmapper is business relationships.
- ralusek 10y agoMind giving a quick description of getting PhantomJS to work?