6 ms·

i like the function. saved me a lot of manual work already ;-) just use it when you need it and be aware of the downsides...

boring_twenties 6y ago
Uh, k? What does being aware of the downsides get you? Is there some alternate interface that avoids the downsides?
- mpfundstein 6y ago
  I am just saying that the function has merit. Even if it could have been written much better. Thats all :-) It served me well. Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?
  - boring_twenties 6y ago
    So just what the fuck are you supposed to use on a directory with 1000 entries?
    - icholy 6y ago
      You can drop down into the lower level apis https://godoc.org/golang.org/x/sys/unix https://godoc.org/golang.org/x/sys/unix
      - doteka 6y ago
        Yeah, nope. There are many appropriate levels of abstraction in between “do this the dumbest way possible” and “write raw unix syscalls using a library outside of the standard distribution”.
        icholy 6y ago
        Just because something doesn't work for your use-case doesn't make it the "dumbest way possible". It just makes you sound dumb.
        doteka 6y ago
        No, this is literally the dumbest way possible to accomplish the task. It is as close to a brute force approach as you can get, I can see no way in which it could be less efficient without being malicious. And it is the only way of accomplishing this task in the standard library of a language which prides itself on performance and maintainability. Also, thanks for the ad hominem, perhaps you shouldn’t attach your self worth to a programming language and see how that works out for you.
        icholy 6y ago
        There's https://golang.org/pkg/os/#File.Readdir https://golang.org/pkg/os/#File.Readdir
        Someone 6y ago
        Which, as far as I can tell (couldn’t find opendir/closedir), has the same problem. I think they named the call after the posix one. If so, they must have known about opendir/closedir.
        icholy 6y ago
        It uses https://golang.org/pkg/syscall/#ReadDirent https://golang.org/pkg/syscall/#ReadDirent and https://golang.org/pkg/syscall/#ParseDirent https://golang.org/pkg/syscall/#ParseDirent to incrementally read them.
      - boring_twenties 6y ago
        That interface is just awful, here's a byte buffer and some offsets, good luck.
    - hu3 6y ago
      https://github.com/karrick/godirwalk https://github.com/karrick/godirwalk
  - doteka 6y ago
    Let’s not do participation trophies though, Pike and Thompson were the original authors of Go. Both of them should know better. It’s fine to include an inefficient version if you also have a ‘proper’ way to fall back to when performance does matter. And if performance does not matter, why are you even using Go? Funnily enough Python Ruby and Node do expose performant ways to do this in the stdlib, so...
  - deleted 6y ago
    [deleted]
- Mathnerd314 6y ago
  This is the syscall: https://www.man7.org/linux/man-pages/man2/getdents.2.html https://www.man7.org/linux/man-pages/man2/getdents.2.html You opendir, read 100 entries, do whatever with them, read 100 more, repeat, then closedir. So it's a buffered interface similar to normal I/O. There's been proposals for doing readdir+stat in one syscall, but nothing merged yet. https://lwn.net/Articles/606995/ https://lwn.net/Articles/606995/
  - yxhuvud 6y ago
    You can also use io_uring and at least do all the stats in a single syscall (or none, depending on what flags you create the ring with, but for normal use it would be one call).
    - Mathnerd314 6y ago
      Ah yes, forgot about uring. Apparently readdir was "next on the list" at the end of last year: https://twitter.com/mathjock/status/1205996402930855937 https://twitter.com/mathjock/status/1205996402930855937 But I don't see it in the tree. I'm guessing sockets got more priority.
Varriount 6y ago
And what if I'm searching through a large directory? Your response doesn't address the argument being made. Creating an API where the entire set of data is returned from a data source, without an option to limit the result, is bad. It means that, if I have to read a directory with 2000 files (and this is quite possible! - think /bin, etc.) the function will have to allocate a slice with 2000 entries, and then call stat on all of them. Stat is not free, especially on certain filesystems (NFS!), nor is reading directory entries.
- mpfundstein 6y ago
  that's why I wrote: be aware of the downside... It's a mega convenient function if you want to slurp in a few files.
  - ikiris 6y ago
    "ORM is great because it means I have all the data avaliable!"
- sethammons 6y ago
  try millions of files :). You can run out of inodes too :)
heavenlyblue 6y ago
In what universe calling stat on every element of an array is “saving a lot of manual work”?
ecnahc515 6y ago
Yeah, it seems similar to `ioutil` functions which aren't necessarily the best approach in all cases, but are intended to make certain common tasks trivial for many cases.

boring_twenties 6y ago

Uh, k? What does being aware of the downsides get you? Is there some alternate interface that avoids the downsides?

mpfundstein 6y ago
I am just saying that the function has merit. Even if it could have been written much better. Thats all :-) It served me well. Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?
- boring_twenties 6y ago
  So just what the fuck are you supposed to use on a directory with 1000 entries?
  - icholy 6y ago
    You can drop down into the lower level apis https://godoc.org/golang.org/x/sys/unix https://godoc.org/golang.org/x/sys/unix
    - doteka 6y ago
      Yeah, nope. There are many appropriate levels of abstraction in between “do this the dumbest way possible” and “write raw unix syscalls using a library outside of the standard distribution”.
      - icholy 6y ago
        Just because something doesn't work for your use-case doesn't make it the "dumbest way possible". It just makes you sound dumb.
        doteka 6y ago
        No, this is literally the dumbest way possible to accomplish the task. It is as close to a brute force approach as you can get, I can see no way in which it could be less efficient without being malicious. And it is the only way of accomplishing this task in the standard library of a language which prides itself on performance and maintainability. Also, thanks for the ad hominem, perhaps you shouldn’t attach your self worth to a programming language and see how that works out for you.
      - icholy 6y ago
        There's https://golang.org/pkg/os/#File.Readdir https://golang.org/pkg/os/#File.Readdir
        Someone 6y ago
        Which, as far as I can tell (couldn’t find opendir/closedir), has the same problem. I think they named the call after the posix one. If so, they must have known about opendir/closedir.
        icholy 6y ago
        It uses https://golang.org/pkg/syscall/#ReadDirent https://golang.org/pkg/syscall/#ReadDirent and https://golang.org/pkg/syscall/#ParseDirent https://golang.org/pkg/syscall/#ParseDirent to incrementally read them.
    - boring_twenties 6y ago
      That interface is just awful, here's a byte buffer and some offsets, good luck.
  - hu3 6y ago
    https://github.com/karrick/godirwalk https://github.com/karrick/godirwalk
- doteka 6y ago
  Let’s not do participation trophies though, Pike and Thompson were the original authors of Go. Both of them should know better. It’s fine to include an inefficient version if you also have a ‘proper’ way to fall back to when performance does matter. And if performance does not matter, why are you even using Go? Funnily enough Python Ruby and Node do expose performant ways to do this in the stdlib, so...
- deleted 6y ago
  [deleted]
Mathnerd314 6y ago
This is the syscall: https://www.man7.org/linux/man-pages/man2/getdents.2.html https://www.man7.org/linux/man-pages/man2/getdents.2.html You opendir, read 100 entries, do whatever with them, read 100 more, repeat, then closedir. So it's a buffered interface similar to normal I/O. There's been proposals for doing readdir+stat in one syscall, but nothing merged yet. https://lwn.net/Articles/606995/ https://lwn.net/Articles/606995/
- yxhuvud 6y ago
  You can also use io_uring and at least do all the stats in a single syscall (or none, depending on what flags you create the ring with, but for normal use it would be one call).
  - Mathnerd314 6y ago
    Ah yes, forgot about uring. Apparently readdir was "next on the list" at the end of last year: https://twitter.com/mathjock/status/1205996402930855937 https://twitter.com/mathjock/status/1205996402930855937 But I don't see it in the tree. I'm guessing sockets got more priority.

Varriount 6y ago

And what if I'm searching through a large directory? Your response doesn't address the argument being made. Creating an API where the entire set of data is returned from a data source, without an option to limit the result, is bad. It means that, if I have to read a directory with 2000 files (and this is quite possible! - think /bin, etc.) the function will have to allocate a slice with 2000 entries, and then call stat on all of them. Stat is not free, especially on certain filesystems (NFS!), nor is reading directory entries.

mpfundstein 6y ago
that's why I wrote: be aware of the downside... It's a mega convenient function if you want to slurp in a few files.
- ikiris 6y ago
  "ORM is great because it means I have all the data avaliable!"
sethammons 6y ago
try millions of files :). You can run out of inodes too :)

heavenlyblue 6y ago

In what universe calling stat on every element of an array is “saving a lot of manual work”?

ecnahc515 6y ago

Yeah, it seems similar to `ioutil` functions which aren't necessarily the best approach in all cases, but are intended to make certain common tasks trivial for many cases.