5 ms·
There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-i
by davidsojevic 2y ago
There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate https://github.com/lexiforest/curl-impersonate
There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi https://github.com/lexiforest/curl_cffi
- nyanpasu64 2y agoI suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...
- ImHereToVote 1y agoEasy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.
- kelsey978126 1y agoWhy do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.
- simpaticoder 1y agoSecurity measures never prevent all abuse. They raise the cost of abuse above an acceptable threshold. Many things work like this. Cleaning doesn't eliminate dirt, it dilutes the dirt below an acceptable threshold. Same for "repairing" and "defects", and some other pairs of things that escape me atm.
- abofh 1y agoThat's the same argument as CAPTCHA's - as far as I know there are no bots protesting them making their lives harder, but as a human - my life is much harder than it needs to be because things need me to prove I'm a human. Clean for data ingestion usually means complicated for data creation - optimizing for the advertisers has material cash value downstream, but customers are upstream, and making it harder is material too.
- ImHereToVote 1y agoWhat is so hard about running a fragment shader after the site has loaded?
- abofh 1y agoI have to assume /s, but lacking that -- Why can't you just allow `curl`? You need a human for advertising dollars or a poor mechanism of rate limiting. I want to use your service. If you're buying me a fragment shader, I guess that's fine, but I'm feeding it to the dogs, not plugging in your rando hardware in to my web-browser.
- ImHereToVote 1y agoI just want to limit my server to usual human users. If you have JavaScript disabled, you won't be missed. Sorry.
- ImHereToVote 1y agoWe are talking about Curl bots here. How is what you are saying relevant?
- cAtte_ 1y agono, nyanpasu64's comment extended the discussion to general bot detection
- zffr 1y agoCan't a bot just collect a few real tokens and then send those instead of trying to run the shader?
- ImHereToVote 1y agoHow do you automate that? Just generate a new token for each day.
- kevindamm 1y agoreplay attacks are some of the easiest to automate
- ImHereToVote 1y agoBut each client gets their own token to solve.
- gruez 1y agoCan't they use a software renderer like swiftshader? You don't need to pass in an actual gpu through virtio or whatever.
- ImHereToVote 1y agoMaybe you can call a WebGL extension that isn't supported. Or better yet have a couple of overdraws of quads. Their bot will handle it, but it will throttle their CPU like gangbusters.
- gruez 1y agoSounds like a PoW system with extra steps?
- ImHereToVote 1y agoIt's exactly a PoW system, but with fewer steps. Most bots can't run GPU workloads. Some do, and that's fine.
- salawat 1y agoMy hardware is not yours to galavant in, and your fitzing around in my digital home for your server's sake just lays bare you are being self-referentially inconsistent. You value the sanctity of your hardware, not mine. You may not realize it, but you're sounding exactly like part of the problem here.
- ImHereToVote 1y agoI don't think running a fragment to calculate one blit operation isn't doing anything to the "sanctity of your machine". Believe me, Chrome abuses your GPU much more than that. Turn off JavaScript if it bothers you so much. You won't be missed.
- deleted 1y ago[deleted]
- bdhcuidbebe 1y agoYou are just guessing, please stop. Also, you’re wrong. All serious scraping is using browsers today.
- _el1s7 1y agoThere's also a module for fully integrating this with the Python requests library: https://github.com/el1s7/curl-adapter https://github.com/el1s7/curl-adapter
- RKFADU_UOFCCLEL 1y agoAll these "advanced" technologies that change faster than I can turn my neck, to make a simple request that looks like it was one of the "certified" big 3 web browsers, which will ironically tax the server less than a certified browser. Is this the nightmare dystopia I was warned about in the 90's? I wonder if anyone here can name the one company that is responsible for this despite positioning themselves as a good guy open source / hacker community contributor.