8 ms·

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of runni

by cle 2y ago

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyone is only using software approved by them, and to ensure that they can identify you. From multiple angles too (security, bots, DDoS, etc.), and it's not just limited to browsers either. End goal seems to be: prove your identity to the megacorps so they can track everything you do and also ensure you are only doing things they approve of. I think the security arguments are just convenient rationalizations in service of this goal.

throwaway99210 2y ago
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
- berkes 2y ago
  But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to application firewalls and private subnets and whatnot. E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. So, I think the amount of attacks indeed can be insane. But the amount that need over zealous tracking is to be countered, is, AFAICS, rather small.
  - throwaway99210 2y ago
    > E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. the arms race is always escalating
    - JohnMakin 2y ago
      we see the same thing, also with a financial company, the most successful strategies we’ve seen is making stuff like this extremely expensive for whoever it is if we see it, and they stop or slow down to a point it becomes not worth it and they move on. sometimes that’s really all you can do without harming legit traffic.
    - josephcsible 2y ago
      Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]
      - RiverCrochet 2y ago
        That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attempts to access `.env`, for example.
        josephcsible 2y ago
        When legitimate users viewed that forum post, their browsers would, in the course of loading the image, attempt to HTTP GET /phpmyadmin/whatever.png, with that being the URL in the actual request in the header.
- cle 2y ago
  Yep totally agree these are problems. I don't have a good alternative proposal either, I'm just disappointed with what we're converging on.
- code51 2y ago
  Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company. I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is their execution, the angle they attack. If, say, the financial company you work for is capable in both product and technical aspect, I assume it leaves no gap. It's the main place to access the service and all the side benefits.
  - miki123211 2y ago
    > Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value Sometimes the customer you have isn't the customer you want. As a bank, you don't want the customers that will try to log in to 1000 accounts, and then immediately transfer any money they find to the Seychelles. As a ticketing platform, you don't want the customers that buy tickets and then immediately sell them on for 4x the price. As a messaging app, you don't want the customers who have 2000 bot accounts and use AI to send hundreds of thousands of spam messages a day. As a social network, you don't want the customers who want to use your platform to spread pro-russian misinformation. In a sense, those are "customer needs left changing", but neither you nor otherr customers want those needs to be automatible.
- deleted 2y ago
  [deleted]
schnable 2y ago
A lot of the motivation comes from government regulations too. Right now this is mostly in banking, but social media and porn regs are coming too.
- lelandfe 2y ago
  PornHub and all of its affiliate sites now block all residents of Alabama, Arkansas, Idaho, Indiana, Kansas, Kentucky, Mississippi, Montana, Nebraska, North Carolina, Texas, Utah, and Virginia (and Florida on Jan 1): https://www.pcmag.com/news/pornhub-blocked-florida-alabama-texas-virginia-nc-age-verification-vpn https://www.pcmag.com/news/pornhub-blocked-florida-alabama-t... Child safety, as always, was the sugar that made the medicine go down in freedom-loving USA. I imagine these states' approaches will try to move to the federal level after Section 230 dies an ignominious death. Keep an eye out for Free Speech Coalition v. Paxton to hit SCOTUS in January: https://www.oyez.org/cases/2024/23-1122 https://www.oyez.org/cases/2024/23-1122
octocop 2y ago
"I have nothing to hide" will eventually spread to everyone. Very unfortunate.
- cle 2y ago
  I'm in a similar boat but it's more like "I have nothing I can hide". These days I just tell friends & family to assume that nothing they do is private.
- Habgdnv 2y ago
  The answer is simple: I have something to hide. I have many things to hide actually. Nothing of these things is illegal currently but I still have many things to hide. And if I have something to hide - I can be worried about many things.
  - Dilettante_ 2y ago
    "It's not that I have something to hide, there's simply nothing I want to show you."
deadbabe 2y ago
Even if the internet was wide open it’s of little use these days. AI will replace any search you would want to do to find information, the only reason to scour the internet now is for social purposes: finding comments and forums or content from other users, and you don’t really need to be untracked to do all that. A megacorp’s main motivation for tracking your identity is to sell you shit or sell your data to other people who want to sell you things. But if you’re using AI the amount of ads and SEO spam that you have to sift through will dramatically reduce, rendering most of those efforts pointless. And most people aren’t using the internet like in the old days: stumbling across quaint cozy boutique websites made by hobbyists about some favorite topic. People just jump on social platforms and consume content until satisfied. There is no money to be made anymore in mass web scraping at scale with impersonated clients, it’s all been consumed.
userbinator 2y ago
They've been planning this stuff for a long time... https://en.wikipedia.org/wiki/Next-Generation_Secure_Computing_Base https://en.wikipedia.org/wiki/Next-Generation_Secure_Computi... ...and we're seeing the puzzle pieces fall into place. Mandated driver signing, TPMs, and more recently remote attestation. "Security" has always been the excuse --- securing their control over you.
- dwattttt 2y ago
  Another trending thread right now is Pegasus/Predator; as much as it may be a facade, to say MS (or any OS vendor) has no business working on security/secure computing is demonstrably false.
  - userbinator 2y ago
    I have no problem with them working on security in the service of the user. The problem is with them claiming to do that, but instead doing the opposite.
jagged-chisel 2y ago
> … helpful for making HTTP reqs without the overhead of running an entire browser stack For those less informed, add “to impersonate the fingerprints of a browser.” One can, obviously, make requests without a browser stack.
matheusmoreira 2y ago
You are on point. There is no open internet without computing freedom. Computers used to be empowering. Cryptography used to be empowering. Then these corporations started using both against us. They own the computers now. Hardware cryptography ensures the computers only run their software now, software that does their the corporation's bidding and enforces their controls. And if we somehow gain control of the computer we are denied every service and essentially ostracized. I don't think it will be long before we are banned from the internet proper for using "unauthorized" devices. It's an incredibly depressing state of affairs. Everything the word "hacker" ever stood for is pretty much dying. It feels like there's no way out.
choeger 2y ago
> ensure you are only doing things they approve of Absolutely. They might not care about individuals, though. It's their approach to shape "markets". The Apple, Google, Amazon, and Microsoft tax is not inevitable and that's their problem. They will fight toe and nail to keep you locked in, call it "innovation", and even cooperate with governments (which otherwise are their natural enemy in the fight for digital control). It's the people that a) don't care much and b) don't have any options. In the end, a large share of our wealth is just pulled from us to these ever more ridiculous rent seeking schemes.
zouhair 2y ago
The disappearance of the third space is killing us.
1vuio0pswjnm7 2y ago
https://github.com/lexiforest/curl_cffi/releases/expanded_assets/v0.8.1b8 https://github.com/lexiforest/curl_cffi/releases/expanded_as...

by cle 2y ago

throwaway99210 2y ago

> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane

berkes 2y ago
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to application firewalls and private subnets and whatnot. E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. So, I think the amount of attacks indeed can be insane. But the amount that need over zealous tracking is to be countered, is, AFAICS, rather small.
- throwaway99210 2y ago
  > E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. the arms race is always escalating
  - JohnMakin 2y ago
    we see the same thing, also with a financial company, the most successful strategies we’ve seen is making stuff like this extremely expensive for whoever it is if we see it, and they stop or slow down to a point it becomes not worth it and they move on. sometimes that’s really all you can do without harming legit traffic.
  - josephcsible 2y ago
    Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]
    - RiverCrochet 2y ago
      That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attempts to access `.env`, for example.
      - josephcsible 2y ago
        When legitimate users viewed that forum post, their browsers would, in the course of loading the image, attempt to HTTP GET /phpmyadmin/whatever.png, with that being the URL in the actual request in the header.
cle 2y ago
Yep totally agree these are problems. I don't have a good alternative proposal either, I'm just disappointed with what we're converging on.
code51 2y ago
Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company. I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is their execution, the angle they attack. If, say, the financial company you work for is capable in both product and technical aspect, I assume it leaves no gap. It's the main place to access the service and all the side benefits.
- miki123211 2y ago
  > Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value Sometimes the customer you have isn't the customer you want. As a bank, you don't want the customers that will try to log in to 1000 accounts, and then immediately transfer any money they find to the Seychelles. As a ticketing platform, you don't want the customers that buy tickets and then immediately sell them on for 4x the price. As a messaging app, you don't want the customers who have 2000 bot accounts and use AI to send hundreds of thousands of spam messages a day. As a social network, you don't want the customers who want to use your platform to spread pro-russian misinformation. In a sense, those are "customer needs left changing", but neither you nor otherr customers want those needs to be automatible.
deleted 2y ago
[deleted]

schnable 2y ago

A lot of the motivation comes from government regulations too. Right now this is mostly in banking, but social media and porn regs are coming too.

lelandfe 2y ago
PornHub and all of its affiliate sites now block all residents of Alabama, Arkansas, Idaho, Indiana, Kansas, Kentucky, Mississippi, Montana, Nebraska, North Carolina, Texas, Utah, and Virginia (and Florida on Jan 1): https://www.pcmag.com/news/pornhub-blocked-florida-alabama-texas-virginia-nc-age-verification-vpn https://www.pcmag.com/news/pornhub-blocked-florida-alabama-t... Child safety, as always, was the sugar that made the medicine go down in freedom-loving USA. I imagine these states' approaches will try to move to the federal level after Section 230 dies an ignominious death. Keep an eye out for Free Speech Coalition v. Paxton to hit SCOTUS in January: https://www.oyez.org/cases/2024/23-1122 https://www.oyez.org/cases/2024/23-1122

octocop 2y ago

"I have nothing to hide" will eventually spread to everyone. Very unfortunate.

cle 2y ago
I'm in a similar boat but it's more like "I have nothing I can hide". These days I just tell friends & family to assume that nothing they do is private.
Habgdnv 2y ago
The answer is simple: I have something to hide. I have many things to hide actually. Nothing of these things is illegal currently but I still have many things to hide. And if I have something to hide - I can be worried about many things.
- Dilettante_ 2y ago
  "It's not that I have something to hide, there's simply nothing I want to show you."

deadbabe 2y ago

Even if the internet was wide open it’s of little use these days. AI will replace any search you would want to do to find information, the only reason to scour the internet now is for social purposes: finding comments and forums or content from other users, and you don’t really need to be untracked to do all that. A megacorp’s main motivation for tracking your identity is to sell you shit or sell your data to other people who want to sell you things. But if you’re using AI the amount of ads and SEO spam that you have to sift through will dramatically reduce, rendering most of those efforts pointless. And most people aren’t using the internet like in the old days: stumbling across quaint cozy boutique websites made by hobbyists about some favorite topic. People just jump on social platforms and consume content until satisfied. There is no money to be made anymore in mass web scraping at scale with impersonated clients, it’s all been consumed.

userbinator 2y ago

They've been planning this stuff for a long time... https://en.wikipedia.org/wiki/Next-Generation_Secure_Computing_Base https://en.wikipedia.org/wiki/Next-Generation_Secure_Computi... ...and we're seeing the puzzle pieces fall into place. Mandated driver signing, TPMs, and more recently remote attestation. "Security" has always been the excuse --- securing their control over you.

dwattttt 2y ago
Another trending thread right now is Pegasus/Predator; as much as it may be a facade, to say MS (or any OS vendor) has no business working on security/secure computing is demonstrably false.
- userbinator 2y ago
  I have no problem with them working on security in the service of the user. The problem is with them claiming to do that, but instead doing the opposite.

jagged-chisel 2y ago

> … helpful for making HTTP reqs without the overhead of running an entire browser stack For those less informed, add “to impersonate the fingerprints of a browser.” One can, obviously, make requests without a browser stack.

matheusmoreira 2y ago

You are on point. There is no open internet without computing freedom. Computers used to be empowering. Cryptography used to be empowering. Then these corporations started using both against us. They own the computers now. Hardware cryptography ensures the computers only run their software now, software that does their the corporation's bidding and enforces their controls. And if we somehow gain control of the computer we are denied every service and essentially ostracized. I don't think it will be long before we are banned from the internet proper for using "unauthorized" devices. It's an incredibly depressing state of affairs. Everything the word "hacker" ever stood for is pretty much dying. It feels like there's no way out.

choeger 2y ago

> ensure you are only doing things they approve of Absolutely. They might not care about individuals, though. It's their approach to shape "markets". The Apple, Google, Amazon, and Microsoft tax is not inevitable and that's their problem. They will fight toe and nail to keep you locked in, call it "innovation", and even cooperate with governments (which otherwise are their natural enemy in the fight for digital control). It's the people that a) don't care much and b) don't have any options. In the end, a large share of our wealth is just pulled from us to these ever more ridiculous rent seeking schemes.

zouhair 2y ago

The disappearance of the third space is killing us.

1vuio0pswjnm7 2y ago

https://github.com/lexiforest/curl_cffi/releases/expanded_assets/v0.8.1b8 https://github.com/lexiforest/curl_cffi/releases/expanded_as...