5 ms·

> I think there’s literally no way for the backend to tell that the request isn’t sent by my browser and is actually being sent by a random Python program. Oh

> I think there’s literally no way for the backend to tell that the request isn’t sent by my browser and is actually being sent by a random Python program. Oh my sweet summer child. Unfortunately, there is a whole industry built around this. This a great blog discussing different detection methods: https://incolumitas.com/ https://incolumitas.com/

vincentmarle 5y ago
Browser calls (and sessions) are indeed tricky to emulate - you'll generally have much better luck with reverse engineering mobile client API calls.
- boilerupnc 5y ago
  Totally agree. I used to work on a load testing product that spent many, many dev hours attempting to achieve a high degree of fidelity on web recordings at the HTTP and sometimes even the socket level of emulation. It was extremely tricky. We employed alot of regex matching mechanisms and used to keep a regression test bucket of thousands of example HTTP traffic recordings to avoid messing up cookies, headers, post data and query strings to name a few things. In the early days, the developer abuses around ASP view state payload were an absolute nightmare to deal with. I used to half-joke that I could speak HTTP after staring at the raw traffic and how 5 page loads could generate 100+ requests which had dependencies on one another. Interestingly, there were also an interesting class of client-server bugs that only were obvious in recordings (e.g. multiple repeat HTTP head requests to check if a resource existed). Each object or library dev clearly had no knowledge that the function triggered just before also wanted to check if that resource exists. This resulted in a huge amount of redundant unnecessary calls because nobody coordinated and optimized at this level. Fun stroll down memory lane.
- aryamaan 5y ago
  any resources for that just like this post? I am going to google as well, but wanted something if people already had on the top of their head
  - vincentmarle 5y ago
    Try the iOS Charles proxy app, it allows you to scan and inspect network traffic for most apps.
paxys 5y ago
The point still stands. The server can use any number of heuristics to try and figure out traffic source but (1) it is still an approximation, since they can all be spoofed, and (2) the more strict you make the detection logic, the more regular users are affected as well.
djeikyb 5y ago
There's a way to make your point without being rude and infantilizing.
- Spivak 5y ago
  I think that interpretation is more on you than anything -- "sweet summer child" is not literally referring to a child but someone whose innocence or blissful ignorance hasn't been ruined by the can of worms they just opened.
  - zamadatix 5y ago
    It means those things precisely because the person is a child. In A Song of Ice and Fire winter can last over a decade hence "summer child", a young one that has never experienced the hardship of winter https://en.wiktionary.org/wiki/sweet_summer_child https://en.wiktionary.org/wiki/sweet_summer_child Regardless, starting out with any variation of "you're blissfully ignorant" isn't needed either. I get offense usually isn't intended but the use of that phrase has always stuck me as a very condescending way to respond.
    - 0xbadcafebee 5y ago
      > I get offense usually isn't intended but the use of that phrase has always stuck me as a very condescending way to respond. You recognize it's not meant to be offensive, but you want to treat it as offensive? Wouldn't it feel better to assume good faith and not treat it as offensive? The same phrase spoken from one individual to another can be interpreted as offensive or not. Tom might say "Fuck you!" to Bill, but Bill is best friends with Tom and knows he's kidding, and laughs. If they didn't know each other, Bill would get offended. But either way it's just words. Tom's intended meaning doesn't change despite what Bill wants to assume the meaning was. It's up to us to decide to get offended or not.
    - evan_ 5y ago
      that definition misplaces the origin of the phrase, "sweet summer's child" has been in use since the mid-19th century or earlier: https://books.google.co.uk/books?id=mGQSAAAAYAAJ&pg=PA52#v=onepage&q&f=false https://books.google.co.uk/books?id=mGQSAAAAYAAJ&pg=PA52#v=o...
Dunedan 5y ago
Her example code doesn't even set a user agent header, making it trivial to distinguish these requests from ones an actual browser would make.
- MaxDPS 5y ago
  Yup, she is using the Requests library which has a default header that explicitly states the request is coming from Python Requests library.
tshaddox 5y ago
Well, in this case the existence of a whole industry built around this is a reflection of the fact that it's impossible to implement a single static solution to the problem. It will forever be a cat and mouse game.
- benmmurphy 5y ago
  Yes. And I think the great thing about her observation is the naive view is actually true in practice (to some degree...). It is very difficult to distinguish between a 'legitimate' user that is a human and a bot. At least without reverting to captcha's and if the benefit to subverting the captcha's is high enough then you can just get human's to solve the captchas.
naniwaduni 5y ago
Perhaps the fact that there's a whole industry built around this instead of, you know, a couple of well-cooked solutions that everyone uses, implies that it's not possible and isn't going to be for the foreseeable future.

vincentmarle 5y ago

Browser calls (and sessions) are indeed tricky to emulate - you'll generally have much better luck with reverse engineering mobile client API calls.

boilerupnc 5y ago
Totally agree. I used to work on a load testing product that spent many, many dev hours attempting to achieve a high degree of fidelity on web recordings at the HTTP and sometimes even the socket level of emulation. It was extremely tricky. We employed alot of regex matching mechanisms and used to keep a regression test bucket of thousands of example HTTP traffic recordings to avoid messing up cookies, headers, post data and query strings to name a few things. In the early days, the developer abuses around ASP view state payload were an absolute nightmare to deal with. I used to half-joke that I could speak HTTP after staring at the raw traffic and how 5 page loads could generate 100+ requests which had dependencies on one another. Interestingly, there were also an interesting class of client-server bugs that only were obvious in recordings (e.g. multiple repeat HTTP head requests to check if a resource existed). Each object or library dev clearly had no knowledge that the function triggered just before also wanted to check if that resource exists. This resulted in a huge amount of redundant unnecessary calls because nobody coordinated and optimized at this level. Fun stroll down memory lane.
aryamaan 5y ago
any resources for that just like this post? I am going to google as well, but wanted something if people already had on the top of their head
- vincentmarle 5y ago
  Try the iOS Charles proxy app, it allows you to scan and inspect network traffic for most apps.

paxys 5y ago

The point still stands. The server can use any number of heuristics to try and figure out traffic source but (1) it is still an approximation, since they can all be spoofed, and (2) the more strict you make the detection logic, the more regular users are affected as well.

djeikyb 5y ago

There's a way to make your point without being rude and infantilizing.

Spivak 5y ago
I think that interpretation is more on you than anything -- "sweet summer child" is not literally referring to a child but someone whose innocence or blissful ignorance hasn't been ruined by the can of worms they just opened.
- zamadatix 5y ago
  It means those things precisely because the person is a child. In A Song of Ice and Fire winter can last over a decade hence "summer child", a young one that has never experienced the hardship of winter https://en.wiktionary.org/wiki/sweet_summer_child https://en.wiktionary.org/wiki/sweet_summer_child Regardless, starting out with any variation of "you're blissfully ignorant" isn't needed either. I get offense usually isn't intended but the use of that phrase has always stuck me as a very condescending way to respond.
  - 0xbadcafebee 5y ago
    > I get offense usually isn't intended but the use of that phrase has always stuck me as a very condescending way to respond. You recognize it's not meant to be offensive, but you want to treat it as offensive? Wouldn't it feel better to assume good faith and not treat it as offensive? The same phrase spoken from one individual to another can be interpreted as offensive or not. Tom might say "Fuck you!" to Bill, but Bill is best friends with Tom and knows he's kidding, and laughs. If they didn't know each other, Bill would get offended. But either way it's just words. Tom's intended meaning doesn't change despite what Bill wants to assume the meaning was. It's up to us to decide to get offended or not.
  - evan_ 5y ago
    that definition misplaces the origin of the phrase, "sweet summer's child" has been in use since the mid-19th century or earlier: https://books.google.co.uk/books?id=mGQSAAAAYAAJ&pg=PA52#v=onepage&q&f=false https://books.google.co.uk/books?id=mGQSAAAAYAAJ&pg=PA52#v=o...

Dunedan 5y ago

Her example code doesn't even set a user agent header, making it trivial to distinguish these requests from ones an actual browser would make.

MaxDPS 5y ago
Yup, she is using the Requests library which has a default header that explicitly states the request is coming from Python Requests library.

tshaddox 5y ago

Well, in this case the existence of a whole industry built around this is a reflection of the fact that it's impossible to implement a single static solution to the problem. It will forever be a cat and mouse game.

benmmurphy 5y ago
Yes. And I think the great thing about her observation is the naive view is actually true in practice (to some degree...). It is very difficult to distinguish between a 'legitimate' user that is a human and a bot. At least without reverting to captcha's and if the benefit to subverting the captcha's is high enough then you can just get human's to solve the captchas.

naniwaduni 5y ago

Perhaps the fact that there's a whole industry built around this instead of, you know, a couple of well-cooked solutions that everyone uses, implies that it's not possible and isn't going to be for the foreseeable future.