9 ms·

File System Interfaces for Go – Draft Design

176 points by networkimprov 6y ago

justicezyx 6y ago
One thing I guess people dont realize is that a lot of Go design is closest resembles Google's internal C++ APIs. There is no exception for this API either. I kind of feel weird when there has been loud praise on Go team, while in general it was no mention of the decade of hard work of evolving C++ APIs...
- paedubucher 6y ago
  Maybe Ken Thompson and Rob Pike worked on those C++ APIs, too. They certainly don't like C++, but also had to used it, and created Go out of frustration.
- jeffbee 6y ago
  Do people really sing the praises of File? Or, do they curse its many and dangerous rough edges? Like you I read this proposal through that lens, but I actually don't see a line from Google's File abstraction to this. Is this Go abstraction really forward-compatible with cloud-native filesystems? And when I say forward compatible I mean would it work with Google's decade-old Colossus? I think it isn't because it puts Stat in the required interface and Stat returns FileInfo, many of the fields of which might be meaningless in a cloud filesystem (like mode, which is a unixism, and IsDir, which doesn't make sense for non-hierarchical filesystems). If you can't tell I'm not much of a fan of trying to abstract over filesystems. Mostly they don't resemble each other at all, except in some extremely high level concepts.
  - justicezyx 6y ago
    So first, what exactly I was meant to say, is that: Go design resembles closest to Google's internal C++ APIs. Then, when I saw people praises Go APIs, I feel that some people worked on C++ APIs at Google, were missing some credits. That's a derived feeling from the above. That is not to say that this File design was already being praised (not to say that it does not deserve praise); or that it should be criticized (not to say that it does not have problems). As for your example to contradict my claim: "Stat returns FileInfo, many of the fields of which might be meaningless in a cloud filesystem": 1. This has to be there because a File API has to be compatible with OS files (Unixy). And Google's API also does the same thing. 2. Incompatibility derives from enforcing information that not present in another scenario. Clearly, the presence of such attributes does not prevent them to be applied on cloud files systems. And further, Cloud file systems do have mode, and directory...
    - athrun 6y ago
      > And further, Cloud file systems do have mode, and directory... The most popular cloud file system, S3, doesn't have folders. It has prefixes, but if you treat them like folders, you're in for a lot of pain.
      - jeffbee 6y ago
        Colossus doesn't have directories either. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke... Page 9
      - justicezyx 6y ago
        What are the pains when treating prefix as directories? I am not familiar with S3.
  - boomlinde 6y ago
    IsDir can return false for filesystems that don't have that concept, which would of course be the correct value in that case. Mode and ModTime are more concerning, but practically, FileMode(0) and time.Time{} can be used respectively in implementations that have no use for them. In terms of the ideal of preferring types as a means to constrain functionality, I think FileInfo fields should have been expressed as granularly as possible using extension interfaces in this case, but practically I'd prefer putting some placeholder values in FileInfo for missing functionality over the clunky way these extensions have to be implemented in in Go.
parhamn 6y ago
Maybe they can just put in double-glob (recursive) support in one Go as the embedded files stuff needs it too. Wonder how that will work from a backwards compatibility perspective? Do asterisks have to be escaped to begin with?
henvic 6y ago
The best thing about Go is how things are designed thinking ahead in the future and avoiding hypes. Instead of a language with more features than one can count, we have something really concise, yet powerful. This is something I miss a lot about Go and I think can perhaps even have a greater impact than generics for most people (at least I feel like this is my case), yet this is only being considered now after ideas matured in the community and people developed ways around it (see https://github.com/golang/go/issues/35950 https://github.com/golang/go/issues/35950). The outcome will probably be something that will be robust and stable and once in the language will likely last a long time unchanged without feeling awkward.
- danudey 6y ago
  I'm viewing this from outside of the ecosystem, but nothing I've seen from Go externally shows any sort of "thinking ahead", so much as "building what Google needs and wants to have". This blog post is just one example of those kinds of issues, and it's just the most recent of such that I've read: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-... Things like no package management, no vendoring, importing modules directly from GitHub without any concept of versioning, the confusing mess that is $GOPATH, the awkward handling of errors, and so on. Lots of things that were fixed, but shouldn't have been broken in the first place. Go certainly has use and functionality and benefits (goroutines sound awesome), but Go itself seems like kind of an awkward mess when viewed outside of the context of "internal Google tool".
  - YesThatTom2 6y ago
    ITS DIFFERENT THEREFORE IT MAKES ME UNCOMFORTABLE!!! WHY CAN’T EVERYTHING BE LIKE MY FIRST COMPUTER??? WAAAAAAAAAAA
  - mpfundstein 6y ago
    maybe you should try it ... what about that? I bet you will change your mind. Both posts that you have linked are pretty much: "meh... i need s.th. to complain about"-posts. I've never been really affected by the issues they are screaming so loud about and believe me, I did not only write some toy projects [1] go is mega awesome and I love going back to the projects where I use it. It's a simple language that makes it very easy to write good code. And the library ecosystem is pretty good as well, especially when it comes to networking stuff. [1] https://github.com/MarkusPfundstein/taylor https://github.com/MarkusPfundstein/taylor
boring_twenties 6y ago
os.Readdir has to be one of the stupidest, most broken interfaces I've ever seen. Reading the whole dir into an array is bad enough as it is, but they even call stat() on every file too.
- mpfundstein 6y ago
  i like the function. saved me a lot of manual work already ;-) just use it when you need it and be aware of the downsides...
  - boring_twenties 6y ago
    Uh, k? What does being aware of the downsides get you? Is there some alternate interface that avoids the downsides?
    - mpfundstein 6y ago
      I am just saying that the function has merit. Even if it could have been written much better. Thats all :-) It served me well. Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?
      - boring_twenties 6y ago
        So just what the fuck are you supposed to use on a directory with 1000 entries?
        icholy 6y ago
        You can drop down into the lower level apis https://godoc.org/golang.org/x/sys/unix https://godoc.org/golang.org/x/sys/unix
        doteka 6y ago
        Yeah, nope. There are many appropriate levels of abstraction in between “do this the dumbest way possible” and “write raw unix syscalls using a library outside of the standard distribution”.
zemnmez 6y ago
I really like the way these proposals preserve file metadata
didip 6y ago
I am surprised that it doesn't have public interfaces with context. Network file system could use the same interfaces + timeout settings.
- icholy 6y ago
  There was discussion about this. One approach would be to tie the entire fs.FS instance to a context. Something akin to `http.Request.WithContext`.
  - jeffbee 6y ago
    How would that be helpful? A context deadline is a fixed point in time. You really want the context (and its deadline) to be specified at the operation scope, not at the file or at the entire filesystem level. And you might have a deadline on a read but be willing to wait forever for a close, so you don't really want the deadline to be attached to the file handle, either.
    - q3k 6y ago
      As long as something like `(f FS) WithContext(ctx context.Context) FS` exists and has overwrite/replace semantics, then that could be used to do more narrow scoping of particular requests, no? Ie. `fs.WithContext(ctxT1).Read()` vs `fs.WithContext(ctxT2).Close()`.
    - andrewstuart2 6y ago
      It definitely wouldn't need to be at the top level. Remember, this is an interface, so func (f SomeConcreteFS) WithContext(ctx context.Context) fs.FS could hold a reference to the underlying filesystem and call appropriate read deadlines, timeouts, etc.
      - sudhirj 6y ago
        And the draft specifically mentions that those methods are the minimum interface - a library that supports contexts could always do this either way.
bfuclusion 6y ago
Is stat really something you want as an abstraction over all filesystems? You might not even have that information available in all of them.
- heinrichhartman 6y ago
  Calling stat on all the files is just really expensive if your directly is large.
  - networkimprov 6y ago
    See the Reddit for an alternative to FileInfo. https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...
  - tptacek 6y ago
    FileInfo is an interface, isn't it? You could populate fields other than "name" lazily, right?
    - TheDong 6y ago
      You can't because the interface doesn't allow any errors. Lazily populating data is usually fallible, and the interface's methods are all infallible.
      - tptacek 6y ago
        OK, I'm convinced. Not a great interface, should just return an iterator.
- skissane 6y ago
  Every filesystem ever has the concept of file attributes. The problem is the exact details of what file attributes are supported vary widely from system to system. The best approach is to support a subset which is reasonably common across platforms – file type[1], modification timestamps, file size in bytes, etc – and an extension mechanism to enable platform-specific attributes. Java NIO handles this reasonably well with the java.nio.file.attribute package[2] in my opinion. (Not sure how easy it would be to port the concepts of that to Go though.) IANA has a registry of OS-specific facts (i.e. file attributes) and OS-specific file types[3] – this is for use of FTP MLST and MLSD commands[4] but the registry is rather empty because that RFC doesn't appear to have got much adoption. It is a good idea though. [1] There is a standard list of file types most platforms support – regular file, directory, link – but there are lots of special file types specific to various platforms (e.g. named pipes, UNIX domain sockets, BSD whiteouts, NTFS junctions), plus some filesystems have different subtypes of regular files or directories. (For example, on IBM z/OS, a "regular file" could be a UNIX file, a VSAM dataset, or a non-VSAM dataset, and the later two both have several subtypes; similarly, z/OS has UNIX directories, but PDS(E) could also be viewed as a non-UNIX type of directory.) [2] https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/nio/file/attribute/package-summary.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... [3] https://www.iana.org/assignments/os-specific-parameters/os-specific-parameters.xhtml https://www.iana.org/assignments/os-specific-parameters/os-s... [4] https://tools.ietf.org/html/rfc3659 https://tools.ietf.org/html/rfc3659
shawnz 6y ago
Much of the criticism I have seen towards Go has been in regard to the poor design of its filesystem APIs (example: https://news.ycombinator.com/item?id=22443363 https://news.ycombinator.com/item?id=22443363). So rethinking this might make the language significantly more attractive for some. Especially considering the addition of generics, I am becoming much more interested in the language again.
- deleted 6y ago
  [deleted]
SEJeff 6y ago
I’m kind of surprised they don’t mention afero at all, who was one of the first big packages to abstract filesystem access. Shoutout as they make unit testing filesystem ops a piece of cake! https://github.com/spf13/afero https://github.com/spf13/afero
- kyrra 6y ago
  They are likely aware of it, as spf13 works on golang at Google. :)
  - SEJeff 6y ago
    Yeah I realize that. It is just surprising is all. It is a very nice to use interface. Getting an equivalent in stdlib would be great.
- shanemhansen 6y ago
  Just a random gopher sharing their level of surprise. I didn't know about the spf13 project, but I instantly thought of https://godoc.org/golang.org/x/tools/godoc/vfs https://godoc.org/golang.org/x/tools/godoc/vfs Looks like they mention that package in their proposal.
- mseepgood 6y ago
  https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/fys9ch1/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...
  - SEJeff 6y ago
    Awesome find, thanks!
LTClipp 6y ago
I would like to put it out there that I'm working on a pathlib library that is attempting to solve a lot of the problems that this design draft is addressing. https://github.com/chigopher/pathlib https://github.com/chigopher/pathlib
networkimprov 6y ago
This (recent but buried) comment suggests replacing FileInfo in the ReadDirFS interface with a DirEntry type, for reasons of performance and future extensibility: https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/g0hwjij/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf... It was prompted by this proposal for os.Readdirentries(): https://github.com/golang/go/issues/40352 https://github.com/golang/go/issues/40352 It seems to me that FileInfo isn't a suitable interface for either a general filesystem construct, or a performant implementation.
sanxiyn 6y ago
This is very valuable for languages to get right. Tcl had it for a long time and I envy them: https://wiki.tcl-lang.org/page/VFS https://wiki.tcl-lang.org/page/VFS
suessflorian 6y ago
Awesome to see some work on filesystem API's but oh boy... you can really see the challenge provided by the backwards compatibility promise. Alias's in os package to definitions in io/fs, the duplication of the http.FileServer.
joshuak 6y ago
This is great. I've had challenges related to filesystem abstraction for years. I scoured the go packages on github for a solution, and found afero[1], but at the time that package had been languishing for years without anyone merging PRs, many of which were critical fixes. It's been picking up again recently so maybe it's better now, but at the time I needed a solution so I built abfs[2]. Abfs is conceptually identical to the go draft design, including the concept of "extension interfaces", although at the fs interface level not the file interface level. One of the problems I ran into early was the need for an easy to use file handle. The `net/http` FileSystem interface shows that you must always implement a custom Open function and a custom File. An object that implements `net/http` File is not adequate, it must actually be cast to http.File. Because of this I've found that it is burdensome to allow flexibility in the definition of the File interface because on the one hand specific functionality not obviously available to a generic File interface must be looked for using type assertion and the consequences of not finding it are poorly defined (should we return an error, should we proceed with some work around, etc), and on the other functions that return a file handle have many possible choices making un-anticipated interoperability unlikely. So instead I opted for always returning a `absfs/absfs` File interface, but return a ErrNotImplemented for functions that are not supported by an implementation. I don't like this, but I like it better then having to do a lot of interface unwrapping to get to the same place, and having it be an error makes the consequences more concrete. Nevertheless, I'm a million percent behind having a filesystem abstraction in the go standard library. It is immediately useful in testing to redirect potentially costly io operations. It is composable, allowing you to do very little work to wrap a file system with mutexes, timouts, and other gating mechanisms. It allows you to support a transactional file system by spawning a FileSystem from another FileSystem. Caching, copy on write, layering, and arbitrary data transformations are all much easier to reason about and implement using a prototypical filesystem as the model. Cheers, thanks for this Russ and Rob! From where I'm standing it would be one of the most valuable improvements to the go ecosystem possible at this point! [1]: https://github.com/spf13/afero https://github.com/spf13/afero [2]: https://github.com/absfs/absfs https://github.com/absfs/absfs

176 points by networkimprov 6y ago

One thing I guess people dont realize is that a lot of Go design is closest resembles Google's internal C++ APIs. There is no exception for this API either. I kind of feel weird when there has been loud praise on Go team, while in general it was no mention of the decade of hard work of evolving C++ APIs...

paedubucher 6y ago
Maybe Ken Thompson and Rob Pike worked on those C++ APIs, too. They certainly don't like C++, but also had to used it, and created Go out of frustration.
jeffbee 6y ago
Do people really sing the praises of File? Or, do they curse its many and dangerous rough edges? Like you I read this proposal through that lens, but I actually don't see a line from Google's File abstraction to this. Is this Go abstraction really forward-compatible with cloud-native filesystems? And when I say forward compatible I mean would it work with Google's decade-old Colossus? I think it isn't because it puts Stat in the required interface and Stat returns FileInfo, many of the fields of which might be meaningless in a cloud filesystem (like mode, which is a unixism, and IsDir, which doesn't make sense for non-hierarchical filesystems). If you can't tell I'm not much of a fan of trying to abstract over filesystems. Mostly they don't resemble each other at all, except in some extremely high level concepts.
- justicezyx 6y ago
  So first, what exactly I was meant to say, is that: Go design resembles closest to Google's internal C++ APIs. Then, when I saw people praises Go APIs, I feel that some people worked on C++ APIs at Google, were missing some credits. That's a derived feeling from the above. That is not to say that this File design was already being praised (not to say that it does not deserve praise); or that it should be criticized (not to say that it does not have problems). As for your example to contradict my claim: "Stat returns FileInfo, many of the fields of which might be meaningless in a cloud filesystem": 1. This has to be there because a File API has to be compatible with OS files (Unixy). And Google's API also does the same thing. 2. Incompatibility derives from enforcing information that not present in another scenario. Clearly, the presence of such attributes does not prevent them to be applied on cloud files systems. And further, Cloud file systems do have mode, and directory...
  - athrun 6y ago
    > And further, Cloud file systems do have mode, and directory... The most popular cloud file system, S3, doesn't have folders. It has prefixes, but if you treat them like folders, you're in for a lot of pain.
    - jeffbee 6y ago
      Colossus doesn't have directories either. http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Keynote.pdf http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke... Page 9
    - justicezyx 6y ago
      What are the pains when treating prefix as directories? I am not familiar with S3.
- boomlinde 6y ago
  IsDir can return false for filesystems that don't have that concept, which would of course be the correct value in that case. Mode and ModTime are more concerning, but practically, FileMode(0) and time.Time{} can be used respectively in implementations that have no use for them. In terms of the ideal of preferring types as a means to constrain functionality, I think FileInfo fields should have been expressed as granularly as possible using extension interfaces in this case, but practically I'd prefer putting some placeholder values in FileInfo for missing functionality over the clunky way these extensions have to be implemented in in Go.

parhamn 6y ago

Maybe they can just put in double-glob (recursive) support in one Go as the embedded files stuff needs it too. Wonder how that will work from a backwards compatibility perspective? Do asterisks have to be escaped to begin with?

henvic 6y ago

The best thing about Go is how things are designed thinking ahead in the future and avoiding hypes. Instead of a language with more features than one can count, we have something really concise, yet powerful. This is something I miss a lot about Go and I think can perhaps even have a greater impact than generics for most people (at least I feel like this is my case), yet this is only being considered now after ideas matured in the community and people developed ways around it (see https://github.com/golang/go/issues/35950 https://github.com/golang/go/issues/35950). The outcome will probably be something that will be robust and stable and once in the language will likely last a long time unchanged without feeling awkward.

danudey 6y ago
I'm viewing this from outside of the ecosystem, but nothing I've seen from Go externally shows any sort of "thinking ahead", so much as "building what Google needs and wants to have". This blog post is just one example of those kinds of issues, and it's just the most recent of such that I've read: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-... Things like no package management, no vendoring, importing modules directly from GitHub without any concept of versioning, the confusing mess that is $GOPATH, the awkward handling of errors, and so on. Lots of things that were fixed, but shouldn't have been broken in the first place. Go certainly has use and functionality and benefits (goroutines sound awesome), but Go itself seems like kind of an awkward mess when viewed outside of the context of "internal Google tool".
- YesThatTom2 6y ago
  ITS DIFFERENT THEREFORE IT MAKES ME UNCOMFORTABLE!!! WHY CAN’T EVERYTHING BE LIKE MY FIRST COMPUTER??? WAAAAAAAAAAA
- mpfundstein 6y ago
  maybe you should try it ... what about that? I bet you will change your mind. Both posts that you have linked are pretty much: "meh... i need s.th. to complain about"-posts. I've never been really affected by the issues they are screaming so loud about and believe me, I did not only write some toy projects [1] go is mega awesome and I love going back to the projects where I use it. It's a simple language that makes it very easy to write good code. And the library ecosystem is pretty good as well, especially when it comes to networking stuff. [1] https://github.com/MarkusPfundstein/taylor https://github.com/MarkusPfundstein/taylor

boring_twenties 6y ago

os.Readdir has to be one of the stupidest, most broken interfaces I've ever seen. Reading the whole dir into an array is bad enough as it is, but they even call stat() on every file too.

mpfundstein 6y ago
i like the function. saved me a lot of manual work already ;-) just use it when you need it and be aware of the downsides...
- boring_twenties 6y ago
  Uh, k? What does being aware of the downsides get you? Is there some alternate interface that avoids the downsides?
  - mpfundstein 6y ago
    I am just saying that the function has merit. Even if it could have been written much better. Thats all :-) It served me well. Reg downsides: Maybe don't use it on a directory with 1000 entries...? Its good to be aware of that or not?
    - boring_twenties 6y ago
      So just what the fuck are you supposed to use on a directory with 1000 entries?
      - icholy 6y ago
        You can drop down into the lower level apis https://godoc.org/golang.org/x/sys/unix https://godoc.org/golang.org/x/sys/unix
        doteka 6y ago
        Yeah, nope. There are many appropriate levels of abstraction in between “do this the dumbest way possible” and “write raw unix syscalls using a library outside of the standard distribution”.

zemnmez 6y ago

I really like the way these proposals preserve file metadata

didip 6y ago

I am surprised that it doesn't have public interfaces with context. Network file system could use the same interfaces + timeout settings.

icholy 6y ago
There was discussion about this. One approach would be to tie the entire fs.FS instance to a context. Something akin to `http.Request.WithContext`.
- jeffbee 6y ago
  How would that be helpful? A context deadline is a fixed point in time. You really want the context (and its deadline) to be specified at the operation scope, not at the file or at the entire filesystem level. And you might have a deadline on a read but be willing to wait forever for a close, so you don't really want the deadline to be attached to the file handle, either.
  - q3k 6y ago
    As long as something like `(f FS) WithContext(ctx context.Context) FS` exists and has overwrite/replace semantics, then that could be used to do more narrow scoping of particular requests, no? Ie. `fs.WithContext(ctxT1).Read()` vs `fs.WithContext(ctxT2).Close()`.
  - andrewstuart2 6y ago
    It definitely wouldn't need to be at the top level. Remember, this is an interface, so func (f SomeConcreteFS) WithContext(ctx context.Context) fs.FS could hold a reference to the underlying filesystem and call appropriate read deadlines, timeouts, etc.
    - sudhirj 6y ago
      And the draft specifically mentions that those methods are the minimum interface - a library that supports contexts could always do this either way.

bfuclusion 6y ago

Is stat really something you want as an abstraction over all filesystems? You might not even have that information available in all of them.

heinrichhartman 6y ago
Calling stat on all the files is just really expensive if your directly is large.
- networkimprov 6y ago
  See the Reddit for an alternative to FileInfo. https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...
- tptacek 6y ago
  FileInfo is an interface, isn't it? You could populate fields other than "name" lazily, right?
  - TheDong 6y ago
    You can't because the interface doesn't allow any errors. Lazily populating data is usually fallible, and the interface's methods are all infallible.
    - tptacek 6y ago
      OK, I'm convinced. Not a great interface, should just return an iterator.
skissane 6y ago
Every filesystem ever has the concept of file attributes. The problem is the exact details of what file attributes are supported vary widely from system to system. The best approach is to support a subset which is reasonably common across platforms – file type[1], modification timestamps, file size in bytes, etc – and an extension mechanism to enable platform-specific attributes. Java NIO handles this reasonably well with the java.nio.file.attribute package[2] in my opinion. (Not sure how easy it would be to port the concepts of that to Go though.) IANA has a registry of OS-specific facts (i.e. file attributes) and OS-specific file types[3] – this is for use of FTP MLST and MLSD commands[4] but the registry is rather empty because that RFC doesn't appear to have got much adoption. It is a good idea though. [1] There is a standard list of file types most platforms support – regular file, directory, link – but there are lots of special file types specific to various platforms (e.g. named pipes, UNIX domain sockets, BSD whiteouts, NTFS junctions), plus some filesystems have different subtypes of regular files or directories. (For example, on IBM z/OS, a "regular file" could be a UNIX file, a VSAM dataset, or a non-VSAM dataset, and the later two both have several subtypes; similarly, z/OS has UNIX directories, but PDS(E) could also be viewed as a non-UNIX type of directory.) [2] https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/nio/file/attribute/package-summary.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... [3] https://www.iana.org/assignments/os-specific-parameters/os-specific-parameters.xhtml https://www.iana.org/assignments/os-specific-parameters/os-s... [4] https://tools.ietf.org/html/rfc3659 https://tools.ietf.org/html/rfc3659

shawnz 6y ago

Much of the criticism I have seen towards Go has been in regard to the poor design of its filesystem APIs (example: https://news.ycombinator.com/item?id=22443363 https://news.ycombinator.com/item?id=22443363). So rethinking this might make the language significantly more attractive for some. Especially considering the addition of generics, I am becoming much more interested in the language again.

deleted 6y ago
[deleted]

SEJeff 6y ago

I’m kind of surprised they don’t mention afero at all, who was one of the first big packages to abstract filesystem access. Shoutout as they make unit testing filesystem ops a piece of cake! https://github.com/spf13/afero https://github.com/spf13/afero

kyrra 6y ago
They are likely aware of it, as spf13 works on golang at Google. :)
- SEJeff 6y ago
  Yeah I realize that. It is just surprising is all. It is a very nice to use interface. Getting an equivalent in stdlib would be great.
shanemhansen 6y ago
Just a random gopher sharing their level of surprise. I didn't know about the spf13 project, but I instantly thought of https://godoc.org/golang.org/x/tools/godoc/vfs https://godoc.org/golang.org/x/tools/godoc/vfs Looks like they mention that package in their proposal.
mseepgood 6y ago
https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/fys9ch1/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf...
- SEJeff 6y ago
  Awesome find, thanks!

LTClipp 6y ago

I would like to put it out there that I'm working on a pathlib library that is attempting to solve a lot of the problems that this design draft is addressing. https://github.com/chigopher/pathlib https://github.com/chigopher/pathlib

networkimprov 6y ago

This (recent but buried) comment suggests replacing FileInfo in the ReadDirFS interface with a DirEntry type, for reasons of performance and future extensibility: https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/g0hwjij/ https://www.reddit.com/r/golang/comments/hv976o/qa_iofs_draf... It was prompted by this proposal for os.Readdirentries(): https://github.com/golang/go/issues/40352 https://github.com/golang/go/issues/40352 It seems to me that FileInfo isn't a suitable interface for either a general filesystem construct, or a performant implementation.

sanxiyn 6y ago

This is very valuable for languages to get right. Tcl had it for a long time and I envy them: https://wiki.tcl-lang.org/page/VFS https://wiki.tcl-lang.org/page/VFS

suessflorian 6y ago

Awesome to see some work on filesystem API's but oh boy... you can really see the challenge provided by the backwards compatibility promise. Alias's in os package to definitions in io/fs, the duplication of the http.FileServer.

joshuak 6y ago

This is great. I've had challenges related to filesystem abstraction for years. I scoured the go packages on github for a solution, and found afero[1], but at the time that package had been languishing for years without anyone merging PRs, many of which were critical fixes. It's been picking up again recently so maybe it's better now, but at the time I needed a solution so I built abfs[2]. Abfs is conceptually identical to the go draft design, including the concept of "extension interfaces", although at the fs interface level not the file interface level. One of the problems I ran into early was the need for an easy to use file handle. The `net/http` FileSystem interface shows that you must always implement a custom Open function and a custom File. An object that implements `net/http` File is not adequate, it must actually be cast to http.File. Because of this I've found that it is burdensome to allow flexibility in the definition of the File interface because on the one hand specific functionality not obviously available to a generic File interface must be looked for using type assertion and the consequences of not finding it are poorly defined (should we return an error, should we proceed with some work around, etc), and on the other functions that return a file handle have many possible choices making un-anticipated interoperability unlikely. So instead I opted for always returning a `absfs/absfs` File interface, but return a ErrNotImplemented for functions that are not supported by an implementation. I don't like this, but I like it better then having to do a lot of interface unwrapping to get to the same place, and having it be an error makes the consequences more concrete. Nevertheless, I'm a million percent behind having a filesystem abstraction in the go standard library. It is immediately useful in testing to redirect potentially costly io operations. It is composable, allowing you to do very little work to wrap a file system with mutexes, timouts, and other gating mechanisms. It allows you to support a transactional file system by spawning a FileSystem from another FileSystem. Caching, copy on write, layering, and arbitrary data transformations are all much easier to reason about and implement using a prototypical filesystem as the model. Cheers, thanks for this Russ and Rob! From where I'm standing it would be one of the most valuable improvements to the go ecosystem possible at this point! [1]: https://github.com/spf13/afero https://github.com/spf13/afero [2]: https://github.com/absfs/absfs https://github.com/absfs/absfs