3 ms·

These statements are so cute. I can reasonably assure you that the distributed k/v store I have written in the thread-per-request style is currently out there

These statements are so cute. I can reasonably assure you that the distributed k/v store I have written in the thread-per-request style is currently out there serving more traffic than you've ever dreamed of. As for your specific points, I disagree totally. Each of those things is easier to me in the synchronous style than in the async style. Certainly something like hedging is way easier.

CyberDildonics 11y ago
Those are some extraordinary claims, but I'm not seeing an extraordinary evidence.
- SamReidHughes 11y ago
  If you're talking to disk, you'll eventually be doing it synchronously anyway, because on Linux the API's for async disk I/O aren't (weren't?) all there yet. For some time I was working on a product with a command line flag to decide between using a worker pool of threads versus event-based disk I/O, and there wasn't a big enough performance difference (if any, I don't remember) for us to keep the event-based code. If you're getting some fancy PCI SSD's or persistent RAM and really want to minimize latency-to-fsync time, maybe things would be different. Also, if you use an event loop, you have to do scheduling yourself. For distributed key/value stores, good thread scheduling is even more important. It removes a big amount of technical risk to lean on the kernel's, unless your product is sufficiently minimalistic. Async will get you real performance advantages for pure networking, but if you're talking to disk it's, when not buggy, not very useful anyway. Also, as for programming style, even if you want event-based syscalls, cooperative userland-managed threads are a big win over callback hell. The only bad thing about them is wasting address-space on 32-bit systems, and maybe just wasting memory too.
  - zzzcpan 11y ago
    Cooperative threads only introduce confusion and are not making anything easier over threads. For example, the following code will fail if one of the functions yields and global array index changes by another cooperative thread: arr_index ++ arr[arr_index].foo = foo() arr[arr_index].bar = bar() There is no protection from this, synchronization primitives are still needed, just like with threads. Which is not the case with event-driven approach. Event-driven code forces you to write asynchronous functions differently from the rest. There is no confusion of what yields where and returns from where. There are just continuations. So, if foo() is asynchronous in the example above, you wouldn't modify global arr_index across callbacks, you either store its value or modify it in a single callback, because you can rely on callbacks to never get interrupted: foo(func(res){ arr_index ++ arr[arr_index].foo = res arr[arr_index].bar = bar() }) Furthermore, because of the very specific constraints on callbacks, event-driven approach allows to implement some sort of composable virtual processes, capable of cleaning up and keeping track of various resources and accepting signals from other virtual processes via callback. And it's pretty easy to do, so some event-driven applications in the wild do that, although not always call them processes. So, again, if you find event-driven programming in any way harder, than multithreaded - you are definitely doing it wrong. Because it completely frees you from thinking about synchronization without adding anything extra that you cannot get used to. As for disks: there is more than one and you are not talking to them directly, there is a filesystem cache in between. Event loops typically use thread pools to fake asynchronous filesystem access and for other work. And it makes some sense latency and performance-wise, because request that waits for some disk does not block requests that want to access other disks, or hit cache, or maybe it's an ssd and accessing it in parallel is faster.
    - SamReidHughes 11y ago
      > Cooperative threads only introduce confusion and are not making anything easier over threads. They introduce (ostensibly or in reality) performance, of the kind that obviates a mass of arguments in favor of event loops. > There is no protection from this, synchronization primitives are still needed, just like with threads. And it's not a problem. > There is no confusion of what yields where and returns from where. This isn't a problem in practice, except when you're (without due care, but it's easy to screw up) migrating from the event-based code that was written with this assumption. You write your code as if it could block anywhere, and then stop wondering whether something could block. If some code is totally not going to block, you use a mutex anyway, hopefully the asserting kind that logs a message if it ever has to wait. It's easy for code written with callbacks to wind up with baked in, undocumented dependencies of chunks of code that aren't allowed to be split up is far worse to maintain than that which lays out that information explicitly. Your example with arr_index is a great demonstration of something trivially solvable by protecting the shared field with a mutex. And if you did that, you would easily notice if foo() or bar() ever decided it should increment arr_index. > Because it completely frees you from thinking about synchronization Well this is untrue. So completely untrue! (Edit: Hell, you very nearly showed it in your own examples. Make the first foo() run your foo(func...) example, make that callback get called without waiting in some cases, and bam, you've got a synchronization problem. Callbacks do nothing to protect you against the class of synchronization problems where the order in which you do stuff matters. For example, naively written rate-limiting code using callbacks will allow incoming actions to pass through out-of-order and might also starve operations that had been blocked. Likewise, you are unprotected from that variable getting incremented when a callback gets called sooner than you'd expect -- and no, on some supposedly high-performance server under load, throwing every callback around the event loop an extra time is not an acceptable option. Ultimately it's moot -- for sane enforcement of proper ordering, you have to rely on some sort of explicit mutual exclusion.) > some sort of composable virtual processes, capable of cleaning up and keeping track of various resources and accepting signals from other virtual processes via callback. And what if I told you that you can keep track of resources and accept signals from, well, anything, in cooperative (or non-cooperative) threads? Because, you can. > As for disks: there is more than one and you are not talking to them directly, there is a filesystem cache in between. Sometimes you are talking to them directly. Event loops use thread pools for disk I/O (instead of the event-based APIs on some platforms) because, well, I don't know what other peoples' motivations were, but I can tell you it's more portable and kernels seem less buggy when you do it that way.

CyberDildonics 11y ago

Those are some extraordinary claims, but I'm not seeing an extraordinary evidence.

SamReidHughes 11y ago
If you're talking to disk, you'll eventually be doing it synchronously anyway, because on Linux the API's for async disk I/O aren't (weren't?) all there yet. For some time I was working on a product with a command line flag to decide between using a worker pool of threads versus event-based disk I/O, and there wasn't a big enough performance difference (if any, I don't remember) for us to keep the event-based code. If you're getting some fancy PCI SSD's or persistent RAM and really want to minimize latency-to-fsync time, maybe things would be different. Also, if you use an event loop, you have to do scheduling yourself. For distributed key/value stores, good thread scheduling is even more important. It removes a big amount of technical risk to lean on the kernel's, unless your product is sufficiently minimalistic. Async will get you real performance advantages for pure networking, but if you're talking to disk it's, when not buggy, not very useful anyway. Also, as for programming style, even if you want event-based syscalls, cooperative userland-managed threads are a big win over callback hell. The only bad thing about them is wasting address-space on 32-bit systems, and maybe just wasting memory too.
- zzzcpan 11y ago
  Cooperative threads only introduce confusion and are not making anything easier over threads. For example, the following code will fail if one of the functions yields and global array index changes by another cooperative thread: arr_index ++ arr[arr_index].foo = foo() arr[arr_index].bar = bar() There is no protection from this, synchronization primitives are still needed, just like with threads. Which is not the case with event-driven approach. Event-driven code forces you to write asynchronous functions differently from the rest. There is no confusion of what yields where and returns from where. There are just continuations. So, if foo() is asynchronous in the example above, you wouldn't modify global arr_index across callbacks, you either store its value or modify it in a single callback, because you can rely on callbacks to never get interrupted: foo(func(res){ arr_index ++ arr[arr_index].foo = res arr[arr_index].bar = bar() }) Furthermore, because of the very specific constraints on callbacks, event-driven approach allows to implement some sort of composable virtual processes, capable of cleaning up and keeping track of various resources and accepting signals from other virtual processes via callback. And it's pretty easy to do, so some event-driven applications in the wild do that, although not always call them processes. So, again, if you find event-driven programming in any way harder, than multithreaded - you are definitely doing it wrong. Because it completely frees you from thinking about synchronization without adding anything extra that you cannot get used to. As for disks: there is more than one and you are not talking to them directly, there is a filesystem cache in between. Event loops typically use thread pools to fake asynchronous filesystem access and for other work. And it makes some sense latency and performance-wise, because request that waits for some disk does not block requests that want to access other disks, or hit cache, or maybe it's an ssd and accessing it in parallel is faster.
  - SamReidHughes 11y ago
    > Cooperative threads only introduce confusion and are not making anything easier over threads. They introduce (ostensibly or in reality) performance, of the kind that obviates a mass of arguments in favor of event loops. > There is no protection from this, synchronization primitives are still needed, just like with threads. And it's not a problem. > There is no confusion of what yields where and returns from where. This isn't a problem in practice, except when you're (without due care, but it's easy to screw up) migrating from the event-based code that was written with this assumption. You write your code as if it could block anywhere, and then stop wondering whether something could block. If some code is totally not going to block, you use a mutex anyway, hopefully the asserting kind that logs a message if it ever has to wait. It's easy for code written with callbacks to wind up with baked in, undocumented dependencies of chunks of code that aren't allowed to be split up is far worse to maintain than that which lays out that information explicitly. Your example with arr_index is a great demonstration of something trivially solvable by protecting the shared field with a mutex. And if you did that, you would easily notice if foo() or bar() ever decided it should increment arr_index. > Because it completely frees you from thinking about synchronization Well this is untrue. So completely untrue! (Edit: Hell, you very nearly showed it in your own examples. Make the first foo() run your foo(func...) example, make that callback get called without waiting in some cases, and bam, you've got a synchronization problem. Callbacks do nothing to protect you against the class of synchronization problems where the order in which you do stuff matters. For example, naively written rate-limiting code using callbacks will allow incoming actions to pass through out-of-order and might also starve operations that had been blocked. Likewise, you are unprotected from that variable getting incremented when a callback gets called sooner than you'd expect -- and no, on some supposedly high-performance server under load, throwing every callback around the event loop an extra time is not an acceptable option. Ultimately it's moot -- for sane enforcement of proper ordering, you have to rely on some sort of explicit mutual exclusion.) > some sort of composable virtual processes, capable of cleaning up and keeping track of various resources and accepting signals from other virtual processes via callback. And what if I told you that you can keep track of resources and accept signals from, well, anything, in cooperative (or non-cooperative) threads? Because, you can. > As for disks: there is more than one and you are not talking to them directly, there is a filesystem cache in between. Sometimes you are talking to them directly. Event loops use thread pools for disk I/O (instead of the event-based APIs on some platforms) because, well, I don't know what other peoples' motivations were, but I can tell you it's more portable and kernels seem less buggy when you do it that way.