5 ms·

Files are IO, which means a lot of waiting. For what reason wouldn't you want to open them concurrently?

by 9rx 8mo ago

mort96 8mo ago
Opening a file is fairly fast (at least if you're on Linux; Windows not so much). Synchronous code is simpler than concurrent code. If processing files sequentially is fast enough, for what reason would you want to open them concurrently?
nasretdinov 8mo ago
For concurrent processing you'd probably do something like splitting the file names into several batches and process those batches sequentially in each goroutine, so it's very much possible that you'd have an exact same loop for the concurrent scenario. P.S. If you have enough files you don't want to try to open them all at once — Go will start creating more and more threads to handle the "blocked" syscalls (open(2) in this case), and you can run out of 10,000 threads too
- win311fwg 8mo ago
  You'd probably have to be doing something pretty unusual to not use a worker queue. Your "P.S." point being a perfect case in point as to why. If you have a legitimate reason for doing something unusual, it is fine to have to use the tools unusually. It serves as a useful reminder that you are purposefully doing something unusual rather than simply making a bad design choice. A good language makes bad design decisions painful.
  - deleted 8mo ago
    [deleted]
  - mort96 8mo ago
    You have now transformed the easy problem of "iterate through some files" into the much more complex problem of either finding a work queue library or writing your own work queue library; and you're baking in the assumption that the only reasonable way to use that work queue is to make each work item exactly one file. What you propose is not a bad solution, but don't come here and pretend it's the only reasonable solution for almost all situations. It's not. Sometimes, you want each work item to be a list of files, if processing one file is fast enough for synchronisation overhead to be significant. Often, you don't have to care so much about the wall clock time your loop takes and it's fast enough to just do sequentially. Sometimes, you're implementing a non-important background task where you intentionally want to only bother one core. None of these are super unusual situations. It is telling that you keep insisting that any solution that's not a one-file-per-work-item work queue is super strange and should be punished by the language's design, when you haven't even responded to my core argument that: sometimes sequential is fast enough.
    - win311fwg 8mo ago
      > It is telling that you keep insisting Keep insisting? What do you mean by that? > when you haven't even responded to my core argument that: sometimes sequential is fast enough. That stands to reason. I wasn't responding to you. The above comment was in reply to nasretdinov.
      - mort96 8mo ago
        Your comment was in reply to nasretdinov, but its fundamental logic ignores what I've been telling you this whole time. You're pretending that the only solution to iterating through files is a work queue and that any solution that does a synchronous open/close for each iteration is fundamentally bad. I have told you why it isn't: you don't always need the performance.
  - nasretdinov 8mo ago
    Using a "work queue", i.e. a channel would still have a for loop like for filename := range workQueue { fp, err := os.Open(filename) if err != nil { ... } defer fp.Close() // do work } Which would have the same exact problem :)
    - win311fwg 8mo ago
      I don't see the problem. for _, filename := range files { queue <- func() { f, _ := os.Open(filename) defer f.Close() } } or more realistically, var group errgroup.Group group.SetLimit(10) for _, filename := range files { group.Go(func() error { f, err := os.Open(filename) if err != nil { return fmt.Errorf("failed to open file %s: %w", filename, err) } defer f.Close() // ... return nil }) } if err := group.Wait(); err != nil { return fmt.Errorf("failed to process files: %w", err) } Perhaps you can elaborate? I did read your code, but it is not clear where the worker queue is. It looks like it ranges over (presumably) a channel of filenames, which is not meaningfully different than ranging over a slice of filenames. That is the original, non-concurrent solution, more or less.
      - mort96 8mo ago
        I think they imagine a solution like this: // Spawn workers for _ := range 10 { go func() { for path := range workQueue { fp, err := os.Open(path) if err != nil { ... } defer fp.Close() // do work } }() } // Iterate files and give work to workers for _, path := range paths { workQueue <- path }
        win311fwg 8mo ago
        Maybe, but why would one introduce coupling between the worker queue and the work being done? That is a poor design. Now we know why it was painful. What is interesting here is that the pain wasn't noticed as a signal that the design was off. I wonder why? We should dive into that topic. I suspect at the heart of it lies why there is so much general dislike for Go as a language, with it being far less forgiving to poor choices than a lot of other popular languages.

Files are IO, which means a lot of waiting. For what reason wouldn't you want to open them concurrently?

by 9rx 8mo ago

Files are IO, which means a lot of waiting. For what reason wouldn't you want to open them concurrently?

Opening a file is fairly fast (at least if you're on Linux; Windows not so much). Synchronous code is simpler than concurrent code. If processing files sequentially is fast enough, for what reason would you want to open them concurrently?

nasretdinov 8mo ago

For concurrent processing you'd probably do something like splitting the file names into several batches and process those batches sequentially in each goroutine, so it's very much possible that you'd have an exact same loop for the concurrent scenario. P.S. If you have enough files you don't want to try to open them all at once — Go will start creating more and more threads to handle the "blocked" syscalls (open(2) in this case), and you can run out of 10,000 threads too

win311fwg 8mo ago
You'd probably have to be doing something pretty unusual to not use a worker queue. Your "P.S." point being a perfect case in point as to why. If you have a legitimate reason for doing something unusual, it is fine to have to use the tools unusually. It serves as a useful reminder that you are purposefully doing something unusual rather than simply making a bad design choice. A good language makes bad design decisions painful.
- deleted 8mo ago
  [deleted]
- mort96 8mo ago
  You have now transformed the easy problem of "iterate through some files" into the much more complex problem of either finding a work queue library or writing your own work queue library; and you're baking in the assumption that the only reasonable way to use that work queue is to make each work item exactly one file. What you propose is not a bad solution, but don't come here and pretend it's the only reasonable solution for almost all situations. It's not. Sometimes, you want each work item to be a list of files, if processing one file is fast enough for synchronisation overhead to be significant. Often, you don't have to care so much about the wall clock time your loop takes and it's fast enough to just do sequentially. Sometimes, you're implementing a non-important background task where you intentionally want to only bother one core. None of these are super unusual situations. It is telling that you keep insisting that any solution that's not a one-file-per-work-item work queue is super strange and should be punished by the language's design, when you haven't even responded to my core argument that: sometimes sequential is fast enough.
  - win311fwg 8mo ago
    > It is telling that you keep insisting Keep insisting? What do you mean by that? > when you haven't even responded to my core argument that: sometimes sequential is fast enough. That stands to reason. I wasn't responding to you. The above comment was in reply to nasretdinov.
    - mort96 8mo ago
      Your comment was in reply to nasretdinov, but its fundamental logic ignores what I've been telling you this whole time. You're pretending that the only solution to iterating through files is a work queue and that any solution that does a synchronous open/close for each iteration is fundamentally bad. I have told you why it isn't: you don't always need the performance.
- nasretdinov 8mo ago
  Using a "work queue", i.e. a channel would still have a for loop like for filename := range workQueue { fp, err := os.Open(filename) if err != nil { ... } defer fp.Close() // do work } Which would have the same exact problem :)
  - win311fwg 8mo ago
    I don't see the problem. for _, filename := range files { queue <- func() { f, _ := os.Open(filename) defer f.Close() } } or more realistically, var group errgroup.Group group.SetLimit(10) for _, filename := range files { group.Go(func() error { f, err := os.Open(filename) if err != nil { return fmt.Errorf("failed to open file %s: %w", filename, err) } defer f.Close() // ... return nil }) } if err := group.Wait(); err != nil { return fmt.Errorf("failed to process files: %w", err) } Perhaps you can elaborate? I did read your code, but it is not clear where the worker queue is. It looks like it ranges over (presumably) a channel of filenames, which is not meaningfully different than ranging over a slice of filenames. That is the original, non-concurrent solution, more or less.
    - mort96 8mo ago
      I think they imagine a solution like this: // Spawn workers for _ := range 10 { go func() { for path := range workQueue { fp, err := os.Open(path) if err != nil { ... } defer fp.Close() // do work } }() } // Iterate files and give work to workers for _, path := range paths { workQueue <- path }
      - win311fwg 8mo ago
        Maybe, but why would one introduce coupling between the worker queue and the work being done? That is a poor design. Now we know why it was painful. What is interesting here is that the pain wasn't noticed as a signal that the design was off. I wonder why? We should dive into that topic. I suspect at the heart of it lies why there is so much general dislike for Go as a language, with it being far less forgiving to poor choices than a lot of other popular languages.