5 ms·

I had a hard time with IOCP and eventuall ditched it because I didn't find a way to avoid creating a separate read buffer for each and every handle. Suppose I

I had a hard time with IOCP and eventuall ditched it because I didn't find a way to avoid creating a separate read buffer for each and every handle. Suppose I want to watch 10K connections, or files, or such. 4K should be a reasonable read buffer size. Do I really need to spend 40M of memory just to read from the handles simultaneously?

kbenson 8y ago
I'm confused (but I'm also not familiar with the API). It sounds like you're saying you want to be able to read from 10k connections simultaneously while allocating less than a byte of buffer space to each, so I assume I'm misunderstanding some aspect of what you're trying to say (or some aspect of the API that puts this into the correct context).
- jstimpfle 8y ago
  No I'm saying I want to read from N handles simultaneously using M buffers, where M <= N. Of course, the OS shouldn't write to one of my read buffers that currently contains other buffered data. But that doesn't mean that it has to be N = M, since that is a waste. I guess M = 10 * number of OS threads should usually be plenty to achieve good performance. But I'm happy to be corrected by people more experienced with systems performance.
  - kbenson 8y ago
    I believe I understand. By not allocating in the api call itself you can pre-allocate your own buffers and keep track of which are in-use and not in-use yourself and use some subset of the total amount that would be allocated in each api call be reusing buffers after a particular connection is done with it. That makes sense.
- zzzcpan 8y ago
  On unixy systems kernel manages buffers by itself, using as little memory as possible, not allocating anything when no data is received, and userspace doesn't need to allocate any memory either until it is notified and decides to non-blockingly read the data from the kernel. Meaning that waiting for data from connections costs nothing, while on windows it requires preallocating buffer per connection. Basically the same problem that synchronous APIs have. The myth that IOCP is elegant has to die.
- asveikau 8y ago
  In the user mode piece, a single thread can get told "these 10K descriptors are ready for you to call read(2) upon them", then that thread can read them all using the same buffer. [In practice you might allocate more depending on what's going on, but this is a crude example.] In the IOCP model you need to allocate space for all read requests upfront, even if they're not writing you any data currently. Kernel mode would have its own buffers, so memory cost per client is not small in this example... except in terms of user-mode buffers.
magicalhippo 8y ago
What would you have the OS do? If you got connections with inbound data, the OS has to store it somewhere. Might as well be in your read buffer right away. If you ask the OS to read 4k from 10k files simultaneously, why are you surprised that the OS wants the buffer to stay around until the read is completed?
- jstimpfle 8y ago
  It should let me choose which buffer I want to use next, from a pool of buffers that I provide. There is no point in having a 1:1 relationship between buffers and read handles.
  - magicalhippo 8y ago
    That doesn't sound very performant having to switch back into userspace just to get another buffer, then back into kernel space to continue completing the read. Though as far as I can see you could get close to what you wish by doing 1 byte reads.
    - jstimpfle 8y ago
      Your snarkiness is not appreciated and I never said you should do constant switching. It's not required. > If you ask the OS to read 4k from 10k files simultaneously There is a difference between between copying inbound data from 10k handles simultaneously (which does require as many buffers) on the one hand, and on the other hand being open to read from any M-sized subset of 10k handles at once, which requires only M buffers. I also never said to have only 4k of (combined) buffer data for 10k handles. "4k should be a reasonable read buffer size" means 4k per buffer. I don't think I've been unclear, but with a tiny bit of good will you could have figured out what I meant anyway.
      - magicalhippo 8y ago
        FWIW I'm not trying to be snarky. > I never said you should do constant switching. It's not required. I didn't see any other way to interpret your "It should let me choose which buffer I want to use next". Though perhaps what you meant was that you want to be able to tell the OS "yo, here's a bunch of buffers, use these to complete my reads"? > [...] and on the other hand being open to read from any M-sized subset of 10k handles at once, which requires only M buffers. I interpret that as saying "yo OS, I want to read 4k each from these 10k files, but only use these M 4k buffers when doing so", where M << 10k. If a request is slow to be fulfilled won't it hog a buffer, reducing the maximum throughput of the fast requests? Why would this be a performance win?
destructionator 8y ago
> Do I really need to spend 40M of memory just to read from the handles simultaneously? of virtual memory... if it isn't being used it can be paged out.
- jstimpfle 8y ago
  Paging? I don't do paging because I value my mental sanity. I don't think there is a good reason to use paging in 2019 for the vast majority of applications, and the rest is probably simply badly designed software. And I only have 8G of RAM like everybody else, 4G of which have probably never been written to. And if you do use paging, it's still 40M wasted disk space without a good reason.
- gpderetta 8y ago
  but at some point you will need it (when the socket becomes ready to read) and it be extremely costly to to bring it back in.

kbenson 8y ago

I'm confused (but I'm also not familiar with the API). It sounds like you're saying you want to be able to read from 10k connections simultaneously while allocating less than a byte of buffer space to each, so I assume I'm misunderstanding some aspect of what you're trying to say (or some aspect of the API that puts this into the correct context).

jstimpfle 8y ago
No I'm saying I want to read from N handles simultaneously using M buffers, where M <= N. Of course, the OS shouldn't write to one of my read buffers that currently contains other buffered data. But that doesn't mean that it has to be N = M, since that is a waste. I guess M = 10 * number of OS threads should usually be plenty to achieve good performance. But I'm happy to be corrected by people more experienced with systems performance.
- kbenson 8y ago
  I believe I understand. By not allocating in the api call itself you can pre-allocate your own buffers and keep track of which are in-use and not in-use yourself and use some subset of the total amount that would be allocated in each api call be reusing buffers after a particular connection is done with it. That makes sense.
zzzcpan 8y ago
On unixy systems kernel manages buffers by itself, using as little memory as possible, not allocating anything when no data is received, and userspace doesn't need to allocate any memory either until it is notified and decides to non-blockingly read the data from the kernel. Meaning that waiting for data from connections costs nothing, while on windows it requires preallocating buffer per connection. Basically the same problem that synchronous APIs have. The myth that IOCP is elegant has to die.
asveikau 8y ago
In the user mode piece, a single thread can get told "these 10K descriptors are ready for you to call read(2) upon them", then that thread can read them all using the same buffer. [In practice you might allocate more depending on what's going on, but this is a crude example.] In the IOCP model you need to allocate space for all read requests upfront, even if they're not writing you any data currently. Kernel mode would have its own buffers, so memory cost per client is not small in this example... except in terms of user-mode buffers.

magicalhippo 8y ago

What would you have the OS do? If you got connections with inbound data, the OS has to store it somewhere. Might as well be in your read buffer right away. If you ask the OS to read 4k from 10k files simultaneously, why are you surprised that the OS wants the buffer to stay around until the read is completed?

jstimpfle 8y ago
It should let me choose which buffer I want to use next, from a pool of buffers that I provide. There is no point in having a 1:1 relationship between buffers and read handles.
- magicalhippo 8y ago
  That doesn't sound very performant having to switch back into userspace just to get another buffer, then back into kernel space to continue completing the read. Though as far as I can see you could get close to what you wish by doing 1 byte reads.
  - jstimpfle 8y ago
    Your snarkiness is not appreciated and I never said you should do constant switching. It's not required. > If you ask the OS to read 4k from 10k files simultaneously There is a difference between between copying inbound data from 10k handles simultaneously (which does require as many buffers) on the one hand, and on the other hand being open to read from any M-sized subset of 10k handles at once, which requires only M buffers. I also never said to have only 4k of (combined) buffer data for 10k handles. "4k should be a reasonable read buffer size" means 4k per buffer. I don't think I've been unclear, but with a tiny bit of good will you could have figured out what I meant anyway.
    - magicalhippo 8y ago
      FWIW I'm not trying to be snarky. > I never said you should do constant switching. It's not required. I didn't see any other way to interpret your "It should let me choose which buffer I want to use next". Though perhaps what you meant was that you want to be able to tell the OS "yo, here's a bunch of buffers, use these to complete my reads"? > [...] and on the other hand being open to read from any M-sized subset of 10k handles at once, which requires only M buffers. I interpret that as saying "yo OS, I want to read 4k each from these 10k files, but only use these M 4k buffers when doing so", where M << 10k. If a request is slow to be fulfilled won't it hog a buffer, reducing the maximum throughput of the fast requests? Why would this be a performance win?

destructionator 8y ago

> Do I really need to spend 40M of memory just to read from the handles simultaneously? of virtual memory... if it isn't being used it can be paged out.

jstimpfle 8y ago
Paging? I don't do paging because I value my mental sanity. I don't think there is a good reason to use paging in 2019 for the vast majority of applications, and the rest is probably simply badly designed software. And I only have 8G of RAM like everybody else, 4G of which have probably never been written to. And if you do use paging, it's still 40M wasted disk space without a good reason.
gpderetta 8y ago
but at some point you will need it (when the socket becomes ready to read) and it be extremely costly to to bring it back in.