4 ms·

Re: If you single-thread the webserver, [network calls] get executed in series Why is the webserver single-threaded then? And even if it were, why can't networ

by tabtab 7y ago

Re: If you single-thread the webserver, [network calls] get executed in series Why is the webserver single-threaded then? And even if it were, why can't network calls be fanned out within it? If they made multi-threaded webservers, would that solve the problem? Re: Worse, each of those threads take up a few megabytes of RAM that can't be freed until the request completes Do you mean user page requests, or RAM related to network calls? If the first, that probably will happen no matter what. I'm trying to sketch little timing diagrams to see where the bottleneck could come from, and cannot see any "general" problem, only problems related to specific architecture choices. But again, I'm not a systems-level software expert. Is there a common concept behind the bottleneck, lots of architecture-specific nuances adding up (OS+WebSrvr+Compiler), or something else?

nostrademons 7y ago
Are you familiar with the difference between blocking & non-blocking I/O calls? Also, have you done any distributed systems work, where the algorithm that services a given request might require computation on different machines? That's background knowledge that's important; it's hard to know where to start without knowing your background.
- tabtab 7y ago
  Yes, but when I ask web-server experts, they can't supply a clear explanation for what happens with a typical web application, such as the famed "shopping cart application". Or if they do, another expert chimes in and says Expert A's explanation is technically wrong, and then Expert B argues over Expert A's knowledge of how web servers/OS's work or could work, and they delve into arguments over minutia about cache size and chip architecture, and/or bad design of C#'s compiler, etc. I open Pandora's Box but can't close it. Thus, I cannot get the bottom of whether heavy async/await usage is to get around quirks in current systems, or solve a general/universal problem regardless of how web-servers/OSs/compilers/chips are built.
  - nostrademons 7y ago
    Most experts on expert programming forums are idiots. That includes this one, but I've at least worked on an application (Google Search) that I'm pretty sure you've used. Most large websites (with hundreds of millions of users and dev teams in the thousands) don't run all their code in the webserver. They farm out requests to a bunch of backend services running on different boxes - databases, app servers, caches, etc. The central problem with this is that you don't know when a request is going to come back. You don't want to have your webserver simply wait around and do nothing while each request finishes; it'll spend 90% of its time doing nothing, which means you need 10x as much hardware to service the same number of users. But if you have lots of requests to backends in flight, you need some way to synchronize the responses and ensure that the webserver executes the right piece of code only when all of the data needed for it is present. Different languages have different approaches to this. The way we used to do it in C++ is to hand-write a big state machine: each time a response comes back, dump it into memory, consult the current state, move to a new state that might be something like COOKIE_FETCH_COMPLETED or USER_FETCH_COMPLETED or RESULTS_LOADED, execute the code associated with that state (which may kick off even more backend requests), and then wait for the next response to come back, which then triggers the next edge in the state machine, and so on. Async/await just gives language-level support for that. When you await a future, the language interprets it as "execute the next statement of this function only when the data in this future is present. And you can compose futures together, so you're awaiting "all of the data in this list of promises is present". It's a way to apply familiar programming-language concepts like variables and statements to code that's not all running on one box. Do you need async/await? Not if your entire system runs on one box. If every time you make a blocking call, it's just going to a database that runs on the same physical machine as the webserver, you're not really losing anything because the same computer has to do the work anyway. I'd bet that a number of "experts" have only worked on systems like this: if they're talking about OSes and cache sizes and chip architecture, they're not even thinking in terms where concurrency and non-blocking I/O is useful. Oftentimes it's very reasonable to build an initial version like this, but the set of profitable niches that can be served by things like a "shopping cart application" is rapidly declining. Unfortunately, the failure mode for a single-box server with blocking I/O is "your server locks up", and that tends to happen right when it gets most popular, and then you have to rewrite everything if you want to switch it to non-blocking async code. That's the other reason async/await is getting popular: this is one of those areas where scaling is a cliff, not a slope, and most people want to avoid cliffs in their business.

by tabtab 7y ago

nostrademons 7y ago

Are you familiar with the difference between blocking & non-blocking I/O calls? Also, have you done any distributed systems work, where the algorithm that services a given request might require computation on different machines? That's background knowledge that's important; it's hard to know where to start without knowing your background.

tabtab 7y ago
Yes, but when I ask web-server experts, they can't supply a clear explanation for what happens with a typical web application, such as the famed "shopping cart application". Or if they do, another expert chimes in and says Expert A's explanation is technically wrong, and then Expert B argues over Expert A's knowledge of how web servers/OS's work or could work, and they delve into arguments over minutia about cache size and chip architecture, and/or bad design of C#'s compiler, etc. I open Pandora's Box but can't close it. Thus, I cannot get the bottom of whether heavy async/await usage is to get around quirks in current systems, or solve a general/universal problem regardless of how web-servers/OSs/compilers/chips are built.
- nostrademons 7y ago
  Most experts on expert programming forums are idiots. That includes this one, but I've at least worked on an application (Google Search) that I'm pretty sure you've used. Most large websites (with hundreds of millions of users and dev teams in the thousands) don't run all their code in the webserver. They farm out requests to a bunch of backend services running on different boxes - databases, app servers, caches, etc. The central problem with this is that you don't know when a request is going to come back. You don't want to have your webserver simply wait around and do nothing while each request finishes; it'll spend 90% of its time doing nothing, which means you need 10x as much hardware to service the same number of users. But if you have lots of requests to backends in flight, you need some way to synchronize the responses and ensure that the webserver executes the right piece of code only when all of the data needed for it is present. Different languages have different approaches to this. The way we used to do it in C++ is to hand-write a big state machine: each time a response comes back, dump it into memory, consult the current state, move to a new state that might be something like COOKIE_FETCH_COMPLETED or USER_FETCH_COMPLETED or RESULTS_LOADED, execute the code associated with that state (which may kick off even more backend requests), and then wait for the next response to come back, which then triggers the next edge in the state machine, and so on. Async/await just gives language-level support for that. When you await a future, the language interprets it as "execute the next statement of this function only when the data in this future is present. And you can compose futures together, so you're awaiting "all of the data in this list of promises is present". It's a way to apply familiar programming-language concepts like variables and statements to code that's not all running on one box. Do you need async/await? Not if your entire system runs on one box. If every time you make a blocking call, it's just going to a database that runs on the same physical machine as the webserver, you're not really losing anything because the same computer has to do the work anyway. I'd bet that a number of "experts" have only worked on systems like this: if they're talking about OSes and cache sizes and chip architecture, they're not even thinking in terms where concurrency and non-blocking I/O is useful. Oftentimes it's very reasonable to build an initial version like this, but the set of profitable niches that can be served by things like a "shopping cart application" is rapidly declining. Unfortunately, the failure mode for a single-box server with blocking I/O is "your server locks up", and that tends to happen right when it gets most popular, and then you have to rewrite everything if you want to switch it to non-blocking async code. That's the other reason async/await is getting popular: this is one of those areas where scaling is a cliff, not a slope, and most people want to avoid cliffs in their business.