7 ms·
Search all Craigslist cities for a remote job
- chovy 8y agoJust enter your search phrase like "node.js" or "javascript"
- yubiox 8y agoMight want to take this out: console.log(items);
- giancarlostoro 8y agoI usually set those to 'debug' instead, so I can filter them back into my JS console: console.debug(items);
- dhritzkiv 8y agoThe API it's calling seems to be overwhelmed at the time of this comment. The console is full of 429 (too many request) errors
- chovy 8y agoshould be good now. i upgraded my plan.
- worg 8y agoIt'd be nice to have a loading indicator.
- x2f10 8y agoThank you for this comment. I considered the webpage dead.
- deleted 8y ago[deleted]
- gerpsh 8y agoDude, you're making ScraperAPI requests (with your API key in plain text as a parameter) directly from the client app...this thing is a nice idea but it'd probably be a good idea to clean it up and repost.
- criddell 8y agoI thought these multi-region searches were against Craigslist's terms of service.
- smt88 8y agoCraigslist is famously aggressive about scraping, too. While helping with some university research, I found that their anti-scraping defenses were really difficult to get around. They've also been litigious about this in the past, so this project would undoubtedly get a cease-and-desist if it ever became a blip on their radar.
- MegaButts 8y agoI'm guessing that's because of AirBnB and how they got their start by essentially stealing posts from Craigslist.
- massivecali 8y agoI thought Padmapper got their start that way, by essentially providing a better apartment hunt interface than CL had at the time.
- rhizome 8y agoAnd they got the banhammer, too. Not too hard to see where the fulcrum of this "settlement" is: https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-settle-suit-craigslist-over-use-real-estate-facts https://www.eff.org/deeplinks/2015/06/padmapper-and-3taps-se...
- 8y ago
- starpilot 8y agonice it doesn't work.
- anonytrary 8y agoI, too, entered a search query, pressed go, and witnessed absolutely nothing happen. Turns out, you just have to wait a bit. It scrapes Craigslist in real time from the client, so you have to wait a bit for results to pop up. It'd be nice if they had a loading/progress indicator.
- chovy 8y agoI'm working on an indicator next.
- point78 8y agoFill us in on your thoughts for not believing you will get sued by craigslist after multiple comments in the thread.
- sithlord 8y agowould this not fall into a similar category of happened to linkedin? The data is publicly facing, and not behind a paywall or a password, therefore, basically fair-use of internet? ianal, nor do I know much about the subject, but seems like if its not at a minimum password protected information, it is pretty fair-use.
- whalesalad 8y agoLooks like we're performing a DDoS against https://api.scraperapi.com https://api.scraperapi.com (the underlying data source) at this point.
- tomca32 8y agoCool idea, but the implementation is kinda crazy. Good enough to just test the idea at home on your own machine, but pretty risky to expose it to the world and let a ton of other people go nuts making bazzilions of requests. Also, exposed API key in the request.
- diminoten 8y agoSpamming 429s in console, oops!
- OrgNet 8y agoPlay with dog remotely? https://i.imgur.com/OebiBWr.png https://i.imgur.com/OebiBWr.png
- hardlianotion 8y agoIt is a very popular request.
- giancarlostoro 8y agoCan you stack up on those and make 5 grand a month?
- point78 8y agoCraigslist will send you a take down notice in a week, this is explicitly against their terms. I'd advise taking it down before a lawsuit.
- ravenstine 8y agoThat's basically what happened to Padmapper, sadly. With the amount of scams causing a decline of their platform, I'd think that Craigslist would want other sources of traffic. I know that most sites have a policy against scraping/indexing, but Craigslist has their head up their own ass.
- rdiddly 8y agoI feel like anybody who quibbles about scraping has their head up there. It's like, are you on the internet or not? Yes? Then you are on the internet.
- biztos 8y agoMaybe timeliness is the important thing to Craigslist. If you scrape all the jobs data as soon as it shows up, and repost it to your site, then maybe employers think you're the one they should be paying, based on prospects coming in your site and not CL. I would assume that matters more than if you scraped older data or got older data by scraping it more slowly.
- momofarm 8y agowhat I see the freelance job is playing with dog..... should there be a playing with cat job lol
- chovy 8y agothosse freelance dog walkers are huge spammers. I've already removed duplicate titles across cities and you still see a ton of them.
- femto113 8y agoI'm kinda curious how telecommuting to this job actually works. https://detroit.craigslist.org/okl/lab/d/asphalt-laborers/6848551616.html https://detroit.craigslist.org/okl/lab/d/asphalt-laborers/68...
- rxhernandez 8y agoRobots probably.
- noitsnot 8y agoWow, I stopped it at 400 requests. I'm not sure how the site hasn't been shut down at this point.
- skizm 8y agoCan you just search Google for “<term> remote site:craigslist.org” for similar results?
- rkho 8y agoYou should really consider caching results and only fetching them again after a certain amount of time has passed since the last time you scraped them. This way you're not blowing through your ScraperAPI limit.
- codesternews 8y agoHmmm. "You're sending requests a bit too fast! Please slow down your requests, or contact support@scraperapi.com with any questions."
- ada1981 8y agoAll you critics are missing the obvious blue ocean market in remote dog entertainment services that clearly is begging to be served.
- fidla 8y agohmm...doesn't work in chrome
- mothsonasloth 8y agoIts cool but the main problem I am facing with searching for remote jobs. Is that most remote US jobs are not technically remote. They are for US citizens only, so I can't work on it from Europe.
- timsayshey 8y agoThis is an amazing project! Probably one of the most useful ways to actually find quality job listings. If only Craigslist would offer site-wide search natively. Btw, you should open-source this on GitHub. You should also include a crypto/paypal donation option.
- okay3637 8y agoi need a job