5 ms·

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google

I dunno, are features like partial file overwrites necessary to make something a filesystem? This reminds me of how there are lots of internal systems at Google whose maintainers keep asserting are not filesystems, but everyone considers them so, to the point where "_____ is not a filesystem" has become an inside joke.

fiddlerwoaroof 3y ago
Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.
- mickael-kerjean 3y ago
  Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } My mum really couldn't care less about the posix semantic as soon as she can see the pictures of my kid which happen to be on S3 [1] https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash
  - wwalexander 3y ago
    Reducing things to basically the interface you laid out is the point of 9p [1], and is what Plan 9’s UNIX-but-distributed design was built on top of. Same inventor as Go! If you haven’t dived down the Plan 9 rabbit hole yet, it’s a beautiful and haunting vision of how simple cloud computing could have been. [1] https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html
  - MrJohz 3y ago
    I think this interface is less interesting than the semantics behind it, particularly when it comes to concurrency: what happens when you delete a folder, and then try and create a file in that folder at the same time? What happens when you move a folder to a new location, and during that move, delete the new or old folders? Like yes, for your mum's use case, with a single user, it's probably not all that important that you cover those edge cases, but every time I've built pseudo-filesystems on top of non-filesystem storage APIs, those sorts of semantic questions have been where all the problems have hidden. It's not particularly hard to implement the interface you've described, but it's very hard to do it in such a way that, for example, you never have dangling files that exist but aren't contained in any folder, or that you never have multiple files with the same path, and so on.
    - mickael-kerjean 3y ago
      All those considerations are important when implementing the interface but the interface itself isn't invalidated by those concerns and can cope with those constraints fine.
- DonHopkins 3y ago
  Can S3 murder your wife like ReiserFS and Reiser4? https://en.wikipedia.org/w/index.php?title=Comparison_of_file_systems&oldid=209063556#Features https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...
CobrastanJorji 3y ago
They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers who are angry that S3 isn't as fast as some real filesystem solution that the customer migrated from because S3 was cheaper. When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.
- albert_e 3y ago
  If a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ... Can we really blame AWS? I am sure none of official AWS documentations or examples show such an architecture. ---- Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS. "S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system. For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.
  - albert_e 3y ago
    *dime a dozen (Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)
karmasimida 3y ago
Exactly, especially when the concept of filesystem really is defined before the whole internet scale becomes a thing or reality. Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?
deleted 3y ago
[deleted]
yencabulator 3y ago
The problem is once you let go of those semantics, a lot of software stops working if run against such a "filesystem". If you dilute the meaning of "filesystem" too much, it becomes less useful as a term. https://en.wikipedia.org/wiki/Andrew_File_System https://en.wikipedia.org/wiki/Andrew_File_System was interesting, I'd actually love to see something similar re-implemented with modern ideas, but it's more of an direct-access archival system than a general-purpose filesystem[1], you can't just put files written by arbitrary software on it. It's a bit like NFS without locks&leases, but even less like a normal filesystem; only really good for files created once that "settle down" into effectively being read-only. [1]: I wrote https://github.com/bazil/plop https://github.com/bazil/plop that is (unfortunately undocumented) content-addressed immutable file storage over object storage, used in conjunction with a git repo with symlinks to it to manage the "naming layer". See https://bazil.org/doc/ https://bazil.org/doc/ for background, plop is basically a simplification of the ideas to get to working code easier. Site hasn't been updated in almost a decade, wow. It's in everyday use though!

fiddlerwoaroof 3y ago

Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.

mickael-kerjean 3y ago
Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface: type IBackend interface { Ls(path string) ([]os.FileInfo, error) Cat(path string) (io.ReadCloser, error) Mkdir(path string) error Rm(path string) error Mv(from string, to string) error Save(path string, file io.Reader) error Touch(path string) error } My mum really couldn't care less about the posix semantic as soon as she can see the pictures of my kid which happen to be on S3 [1] https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash
- wwalexander 3y ago
  Reducing things to basically the interface you laid out is the point of 9p [1], and is what Plan 9’s UNIX-but-distributed design was built on top of. Same inventor as Go! If you haven’t dived down the Plan 9 rabbit hole yet, it’s a beautiful and haunting vision of how simple cloud computing could have been. [1] https://9fans.github.io/plan9port/man/man9/intro.html https://9fans.github.io/plan9port/man/man9/intro.html
- MrJohz 3y ago
  I think this interface is less interesting than the semantics behind it, particularly when it comes to concurrency: what happens when you delete a folder, and then try and create a file in that folder at the same time? What happens when you move a folder to a new location, and during that move, delete the new or old folders? Like yes, for your mum's use case, with a single user, it's probably not all that important that you cover those edge cases, but every time I've built pseudo-filesystems on top of non-filesystem storage APIs, those sorts of semantic questions have been where all the problems have hidden. It's not particularly hard to implement the interface you've described, but it's very hard to do it in such a way that, for example, you never have dangling files that exist but aren't contained in any folder, or that you never have multiple files with the same path, and so on.
  - mickael-kerjean 3y ago
    All those considerations are important when implementing the interface but the interface itself isn't invalidated by those concerns and can cope with those constraints fine.
DonHopkins 3y ago
Can S3 murder your wife like ReiserFS and Reiser4? https://en.wikipedia.org/w/index.php?title=Comparison_of_file_systems&oldid=209063556#Features https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...

CobrastanJorji 3y ago

They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers who are angry that S3 isn't as fast as some real filesystem solution that the customer migrated from because S3 was cheaper. When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.

albert_e 3y ago
If a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ... Can we really blame AWS? I am sure none of official AWS documentations or examples show such an architecture. ---- Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS. "S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system. For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.
- albert_e 3y ago
  *dime a dozen (Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)

karmasimida 3y ago

Exactly, especially when the concept of filesystem really is defined before the whole internet scale becomes a thing or reality. Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?

deleted 3y ago

[deleted]

yencabulator 3y ago

The problem is once you let go of those semantics, a lot of software stops working if run against such a "filesystem". If you dilute the meaning of "filesystem" too much, it becomes less useful as a term. https://en.wikipedia.org/wiki/Andrew_File_System https://en.wikipedia.org/wiki/Andrew_File_System was interesting, I'd actually love to see something similar re-implemented with modern ideas, but it's more of an direct-access archival system than a general-purpose filesystem[1], you can't just put files written by arbitrary software on it. It's a bit like NFS without locks&leases, but even less like a normal filesystem; only really good for files created once that "settle down" into effectively being read-only. [1]: I wrote https://github.com/bazil/plop https://github.com/bazil/plop that is (unfortunately undocumented) content-addressed immutable file storage over object storage, used in conjunction with a git repo with symlinks to it to manage the "naming layer". See https://bazil.org/doc/ https://bazil.org/doc/ for background, plop is basically a simplification of the ideas to get to working code easier. Site hasn't been updated in almost a decade, wow. It's in everyday use though!