5 ms·

Let's say you're opening files upon each loop iteration. If you're not careful you'll run out of open file descriptors before the loop finishes.

mort96 8mo ago
It doesn't just have to be files, FWIW. I once worked in a Go project which used SDL through CGO for drawing. "Widgets" were basically functions which would allocate an SDL surface, draw to it using Cairo, and return it to Go code. That SDL surface would be wrapped in a Go wrapper with a Destroy method which would call SDL_DestroySurface. And to draw a surface to the screen, you need to create an SDL texture from it. If that's all you want to do, you can then destroy the SDL surface. So you could imagine code like this: strings := []string{"Lorem", "ipsum", "dolor", "sit", "amet"} stringTextures := []SDLTexture{} for _, s := range strings { surface := RenderTextToSurface(s) defer surface.Destroy() stringTextures = append(stringTextures, surface.CreateTexture()) } Oops, you're now using way more memory than you need!
- deleted 8mo ago
  [deleted]
- win311fwg 8mo ago
  Why would you allocate/destroy memory in each iteration when you can reuse it to much greater effect? Aside from bad API design, but a language isn't there to paper over bad design decisions. A good language makes bad design decisions painful.
  - mort96 8mo ago
    The surfaces are all of different size, so the code would have to be more complex, resizing some underlying buffer on demand. You'd have to split up the text rendering into an API to measure the text and an API to render the text, so that you could resize the buffer. So you'd introduce quite a lot of extra complexity. And what would be the benefit? You save up to one malloc and free per string you want to render, but text rendering is so demanding it completely drowns out the cost of one allocation.
    - win311fwg 8mo ago
      Why does the buffer need to be resized? Your malloc version allocates a fixed amount of memory on each iteration. You can allocate the same amount of memory ahead of time. If you were dynamically changing the malloc allocation size on each iteration then you have a case for a growable buffer to do the same, but in that case you would already have all the complexity of which you speak as required to support a dynamically-sized malloc.
      - mort96 8mo ago
        The example allocates an SDL_Surface large enough to fit the text string each iteration. Granted, you could do a pre-pass to find the largest string and allocate enough memory for that once, then use that buffer throughout the loop. But again, what do you gain from that complexity?
        win311fwg 8mo ago
        > The example allocates an SDL_Surface large enough to fit the text string each iteration. Impossible without knowing how much to allocate, which you indicate would require adding a bunch of complexity. However, I am willing to chalk that up to being a typo. Given that we are now calculating how much to allocate on each iteration, where is the meaningful complexity? I see almost no difference between: while (next()) { size_t size = measure_text(t); void *p = malloc(size); draw_text(p, t); free(p); } and void *p = NULL; while (next()) { size_t size = measure_text(t); void *p = galloc(p, size); draw_text(p, t); } free(p);
        mort96 8mo ago
        >> The example allocates an SDL_Surface large enough to fit the text string each iteration. > Impossible without knowing how much to allocate But we do know how much to allocate? The implementation of this example's RenderTextToSurface function would use SDL functions to measure the text, then allocate an SDL_Surface large enough, then draw to that surface. > I see almost no difference between: (code example) and (code example) What? Those two code examples aren't even in the same language as the code I showed. The difference would be between the example I gave earlier: stringTextures := []SDLTexture{} for _, str := range strings { surface := RenderTextToSurface(str) defer surface.Destroy() stringTextures = append(stringTextures, surface.CreateTexture()) } and: surface := NewSDLSurface(0, 0) defer surface.Destroy() stringTextures := []SDLTexture{} for _, str := range strings { size := MeasureText(s) if size.X > surface.X || size.Y > surface.Y { surface.Destroy() surface = NewSDLSurface(size.X, size.Y) } surface.Clear() RenderTextToSurface(surface, str) stringTextures = append(stringTextures, surface.CreateTextureFromRegion(0, 0, size.X, size.Y)) } Remember, I'm talking about the API to a Go wrapper around SDL. How the C code would've looked if you wrote it in C is pretty much irrelevant. I have to ask again though, since you ignored me the first time: what do you gain? Text rendering is really really slow compared to memory allocation.
9rx 8mo ago
Files are IO, which means a lot of waiting. For what reason wouldn't you want to open them concurrently?
- mort96 8mo ago
  Opening a file is fairly fast (at least if you're on Linux; Windows not so much). Synchronous code is simpler than concurrent code. If processing files sequentially is fast enough, for what reason would you want to open them concurrently?
- nasretdinov 8mo ago
  For concurrent processing you'd probably do something like splitting the file names into several batches and process those batches sequentially in each goroutine, so it's very much possible that you'd have an exact same loop for the concurrent scenario. P.S. If you have enough files you don't want to try to open them all at once — Go will start creating more and more threads to handle the "blocked" syscalls (open(2) in this case), and you can run out of 10,000 threads too
  - win311fwg 8mo ago
    You'd probably have to be doing something pretty unusual to not use a worker queue. Your "P.S." point being a perfect case in point as to why. If you have a legitimate reason for doing something unusual, it is fine to have to use the tools unusually. It serves as a useful reminder that you are purposefully doing something unusual rather than simply making a bad design choice. A good language makes bad design decisions painful.
    - deleted 8mo ago
      [deleted]
    - mort96 8mo ago
      You have now transformed the easy problem of "iterate through some files" into the much more complex problem of either finding a work queue library or writing your own work queue library; and you're baking in the assumption that the only reasonable way to use that work queue is to make each work item exactly one file. What you propose is not a bad solution, but don't come here and pretend it's the only reasonable solution for almost all situations. It's not. Sometimes, you want each work item to be a list of files, if processing one file is fast enough for synchronisation overhead to be significant. Often, you don't have to care so much about the wall clock time your loop takes and it's fast enough to just do sequentially. Sometimes, you're implementing a non-important background task where you intentionally want to only bother one core. None of these are super unusual situations. It is telling that you keep insisting that any solution that's not a one-file-per-work-item work queue is super strange and should be punished by the language's design, when you haven't even responded to my core argument that: sometimes sequential is fast enough.

mort96 8mo ago

It doesn't just have to be files, FWIW. I once worked in a Go project which used SDL through CGO for drawing. "Widgets" were basically functions which would allocate an SDL surface, draw to it using Cairo, and return it to Go code. That SDL surface would be wrapped in a Go wrapper with a Destroy method which would call SDL_DestroySurface. And to draw a surface to the screen, you need to create an SDL texture from it. If that's all you want to do, you can then destroy the SDL surface. So you could imagine code like this: strings := []string{"Lorem", "ipsum", "dolor", "sit", "amet"} stringTextures := []SDLTexture{} for _, s := range strings { surface := RenderTextToSurface(s) defer surface.Destroy() stringTextures = append(stringTextures, surface.CreateTexture()) } Oops, you're now using way more memory than you need!

deleted 8mo ago
[deleted]
win311fwg 8mo ago
Why would you allocate/destroy memory in each iteration when you can reuse it to much greater effect? Aside from bad API design, but a language isn't there to paper over bad design decisions. A good language makes bad design decisions painful.
- mort96 8mo ago
  The surfaces are all of different size, so the code would have to be more complex, resizing some underlying buffer on demand. You'd have to split up the text rendering into an API to measure the text and an API to render the text, so that you could resize the buffer. So you'd introduce quite a lot of extra complexity. And what would be the benefit? You save up to one malloc and free per string you want to render, but text rendering is so demanding it completely drowns out the cost of one allocation.
  - win311fwg 8mo ago
    Why does the buffer need to be resized? Your malloc version allocates a fixed amount of memory on each iteration. You can allocate the same amount of memory ahead of time. If you were dynamically changing the malloc allocation size on each iteration then you have a case for a growable buffer to do the same, but in that case you would already have all the complexity of which you speak as required to support a dynamically-sized malloc.
    - mort96 8mo ago
      The example allocates an SDL_Surface large enough to fit the text string each iteration. Granted, you could do a pre-pass to find the largest string and allocate enough memory for that once, then use that buffer throughout the loop. But again, what do you gain from that complexity?
      - win311fwg 8mo ago
        > The example allocates an SDL_Surface large enough to fit the text string each iteration. Impossible without knowing how much to allocate, which you indicate would require adding a bunch of complexity. However, I am willing to chalk that up to being a typo. Given that we are now calculating how much to allocate on each iteration, where is the meaningful complexity? I see almost no difference between: while (next()) { size_t size = measure_text(t); void *p = malloc(size); draw_text(p, t); free(p); } and void *p = NULL; while (next()) { size_t size = measure_text(t); void *p = galloc(p, size); draw_text(p, t); } free(p);
        mort96 8mo ago
        >> The example allocates an SDL_Surface large enough to fit the text string each iteration. > Impossible without knowing how much to allocate But we do know how much to allocate? The implementation of this example's RenderTextToSurface function would use SDL functions to measure the text, then allocate an SDL_Surface large enough, then draw to that surface. > I see almost no difference between: (code example) and (code example) What? Those two code examples aren't even in the same language as the code I showed. The difference would be between the example I gave earlier: stringTextures := []SDLTexture{} for _, str := range strings { surface := RenderTextToSurface(str) defer surface.Destroy() stringTextures = append(stringTextures, surface.CreateTexture()) } and: surface := NewSDLSurface(0, 0) defer surface.Destroy() stringTextures := []SDLTexture{} for _, str := range strings { size := MeasureText(s) if size.X > surface.X || size.Y > surface.Y { surface.Destroy() surface = NewSDLSurface(size.X, size.Y) } surface.Clear() RenderTextToSurface(surface, str) stringTextures = append(stringTextures, surface.CreateTextureFromRegion(0, 0, size.X, size.Y)) } Remember, I'm talking about the API to a Go wrapper around SDL. How the C code would've looked if you wrote it in C is pretty much irrelevant. I have to ask again though, since you ignored me the first time: what do you gain? Text rendering is really really slow compared to memory allocation.

9rx 8mo ago

Files are IO, which means a lot of waiting. For what reason wouldn't you want to open them concurrently?

mort96 8mo ago
Opening a file is fairly fast (at least if you're on Linux; Windows not so much). Synchronous code is simpler than concurrent code. If processing files sequentially is fast enough, for what reason would you want to open them concurrently?
nasretdinov 8mo ago
For concurrent processing you'd probably do something like splitting the file names into several batches and process those batches sequentially in each goroutine, so it's very much possible that you'd have an exact same loop for the concurrent scenario. P.S. If you have enough files you don't want to try to open them all at once — Go will start creating more and more threads to handle the "blocked" syscalls (open(2) in this case), and you can run out of 10,000 threads too
- win311fwg 8mo ago
  You'd probably have to be doing something pretty unusual to not use a worker queue. Your "P.S." point being a perfect case in point as to why. If you have a legitimate reason for doing something unusual, it is fine to have to use the tools unusually. It serves as a useful reminder that you are purposefully doing something unusual rather than simply making a bad design choice. A good language makes bad design decisions painful.
  - deleted 8mo ago
    [deleted]
  - mort96 8mo ago
    You have now transformed the easy problem of "iterate through some files" into the much more complex problem of either finding a work queue library or writing your own work queue library; and you're baking in the assumption that the only reasonable way to use that work queue is to make each work item exactly one file. What you propose is not a bad solution, but don't come here and pretend it's the only reasonable solution for almost all situations. It's not. Sometimes, you want each work item to be a list of files, if processing one file is fast enough for synchronisation overhead to be significant. Often, you don't have to care so much about the wall clock time your loop takes and it's fast enough to just do sequentially. Sometimes, you're implementing a non-important background task where you intentionally want to only bother one core. None of these are super unusual situations. It is telling that you keep insisting that any solution that's not a one-file-per-work-item work queue is super strange and should be punished by the language's design, when you haven't even responded to my core argument that: sometimes sequential is fast enough.