4 ms·

In uni I built a simple web scraper in JavaScript. It just ran in the browser. It would fetch a page, extract the links, then fetch those pages. You could watc

by andai 8mo ago

In uni I built a simple web scraper in JavaScript. It just ran in the browser. It would fetch a page, extract the links, then fetch those pages. You could watch it run in realtime and it would build out a tree of nested links as it crawled. It was a lot of fun to watch! (More fun than CLI based crawlers for sure.) The only issue I had was not being able to fetch pages from the front end due to CORS, so I just added a "proxy" to my server in 1 line of PHP. There, now it's secure? ;)

binaryturtle 8mo ago
That's rater like "gnu.org" which blocks you when you're using a slightly older browser. But when you change your user agent to "curl" it magically starts working. Or the German news site "spiegel.de" which also blocks old browsers from accessing the site entirely, unless you change the user agent to "bingbot" (or some other random bot from their whitelist). *insert a facepalm emoji here*
- mrweasel 8mo ago
  They probably do that to keep out the dumbest scrapers and bots. They often present themselves as out of date Chrome browsers.
  - kstrauser 8mo ago
    Nailed it. I get a zillion bit hits claiming to be Safari for iOS 8 and the like. Sorry if you’re trying to visit my site on such a device.
ehutch79 8mo ago
If you know what a user agent is, let alone how to change it, CORS is not meant for you. Its guide rails to help the tech illiterate not get hacked. It raises the bar on what gets through. It’s not going to stop a determined attacker, but will catch enough to make a dent. Defense in depth and all that.
- lxgr 8mo ago
  CORS (or rather the same origin policy, of which CORS is an explicit server-side opt-out) is not a generic security improvement, it solves a very specific problem: (Code on) website A being able to make requests to website B with the cookies of B (often implying user login state/authentication at B) and read the response. In a (possibly better) parallel universe, cross-site requests just don't send cookies or other ambient authentication state like that by default, and we wouldn't need CORS.
- rnghrhfhf 8mo ago
  It sounds like you need to go back to school because you’re entirely
hansvm 8mo ago
CORS protects the browser's knowledge of a user on your website from other websites. It's a big failing of the protocol and/or communications about it that people think it offers any more security guarantees than that.
nisegami 8mo ago
>There, now it's secure? ;) CORS achieved its objective here, because now those requests were coming from your PHP server rather than the user's browser.
patmorgan23 8mo ago
Because the Single Origin Policy is a client side security measure. It's to protect the user from random malicious JS out on the web. If you are writing server side code you're outside the scope of who the security model is trying to protect.
andai 8mo ago
Edit: Guys! I'm talking about cool scraper running in browser (has that ever been done before?), and everyone's hyperfixating on bloody CORS. Well I guess I failed the Show Don't Tell eh :) lost the code when I left uni, I might have to make another one...

by andai 8mo ago

binaryturtle 8mo ago

That's rater like "gnu.org" which blocks you when you're using a slightly older browser. But when you change your user agent to "curl" it magically starts working. Or the German news site "spiegel.de" which also blocks old browsers from accessing the site entirely, unless you change the user agent to "bingbot" (or some other random bot from their whitelist). *insert a facepalm emoji here*

mrweasel 8mo ago
They probably do that to keep out the dumbest scrapers and bots. They often present themselves as out of date Chrome browsers.
- kstrauser 8mo ago
  Nailed it. I get a zillion bit hits claiming to be Safari for iOS 8 and the like. Sorry if you’re trying to visit my site on such a device.

ehutch79 8mo ago

If you know what a user agent is, let alone how to change it, CORS is not meant for you. Its guide rails to help the tech illiterate not get hacked. It raises the bar on what gets through. It’s not going to stop a determined attacker, but will catch enough to make a dent. Defense in depth and all that.

lxgr 8mo ago
CORS (or rather the same origin policy, of which CORS is an explicit server-side opt-out) is not a generic security improvement, it solves a very specific problem: (Code on) website A being able to make requests to website B with the cookies of B (often implying user login state/authentication at B) and read the response. In a (possibly better) parallel universe, cross-site requests just don't send cookies or other ambient authentication state like that by default, and we wouldn't need CORS.
rnghrhfhf 8mo ago
It sounds like you need to go back to school because you’re entirely

hansvm 8mo ago

CORS protects the browser's knowledge of a user on your website from other websites. It's a big failing of the protocol and/or communications about it that people think it offers any more security guarantees than that.

nisegami 8mo ago

>There, now it's secure? ;) CORS achieved its objective here, because now those requests were coming from your PHP server rather than the user's browser.

patmorgan23 8mo ago

Because the Single Origin Policy is a client side security measure. It's to protect the user from random malicious JS out on the web. If you are writing server side code you're outside the scope of who the security model is trying to protect.

andai 8mo ago

Edit: Guys! I'm talking about cool scraper running in browser (has that ever been done before?), and everyone's hyperfixating on bloody CORS. Well I guess I failed the Show Don't Tell eh :) lost the code when I left uni, I might have to make another one...