3 ms·

Sigh. Another article written, it seems, by someone who thinks that all/most software development is on the web. "Storage systems" in the cloud ... it might mak

Sigh. Another article written, it seems, by someone who thinks that all/most software development is on the web. "Storage systems" in the cloud ... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. But this is an illusion likely caused by insufficient native development experierence. The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried). There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.

VyseofArcadia 3y ago
Thank you for phrasing this so much more politely than I would have. I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here. Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".
jeroenhd 3y ago
"POSIX file system for the web" is kind of the whole deal for the company that wrote this. They make something to mount object storage as a filesystem. This isn't a random programmer philosophising about POSIX, it's an ad for a product.
- cuno 3y ago
  Hi, author here. Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
  - rewmie 3y ago
    > (...) leverage the lower cost and higher throughput of object storage like S3 (...) What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
    - mcv 3y ago
      Not to mention that apparently we're supposed to compare that to storing on your local harddisk. Isn't local storage always far higher throughput than accessing something across the internet? Being able to store and access stuff on the internet as if it's local is a great idea, but the way they sell it is unconvincing and sounds like they don't really understand what they're talking about.
      - cuno 3y ago
        Hi author here, sorry I missed this post. The performance benchmarks and cost comparisons are for comparing S3 vs EBS (ext4 formatted), EFS, FSx Lustre and others within the same datacenter (i.e. LAN use case rather than WAN use case). That means if you have an EC2 instance running in, say AWS Ohio, and are comparing those storage options also within AWS Ohio, then cunoFS is both cheaper and higher throughput than those other options. It's a different story over WAN. In that case, your own local NVMe storage is going to be cheaper and generally faster that remote storage over a WAN. But that local NVMe storage (on say your solo laptop) isn't going to have anywhere near the Enterprise-grade redundancy, availability and scalability that AWS S3/Azure Blob/Storj/Wasabi/etc has.
- pdntspa 3y ago
  It smells like a bunch of FUD to me Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone
  - psd1 3y ago
    FUD is common both ways. Your comment could be interpreted as FUD against anything new. I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)
    - pdntspa 3y ago
      Yeah, I use all of those. But what I can't really use is some low-rent sales copy trying to persuade me to abandon FOSS tooling in favor of yet another rent-seeking cloud solution. New != Good
      - psd1 3y ago
        Agreed, with only one slight quibble: content can be good even when the purpose is sales.
jasode 3y ago
>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...] The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried). Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user. So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX That's why the blog post has the following links to POSIX-on-top-of-S3-objects : https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse https://github.com/kahing/goofys https://github.com/kahing/goofys https://www.cuno.io/ https://www.cuno.io/
- cuno 3y ago
  Hi, author here. Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
  - jonhohle 3y ago
    > in fact, in most cases, there is no filesystem at all Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)? There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
calvinmorrison 3y ago
APIs keep getting conflate for REST APIs. As far as I understand it, the stdlib is an api
nvm0n2 3y ago
I don't know. Consider: 1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems). 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner. S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions. But ultimately, they're both storing files. The differences are fairly minor.
- PaulDavisThe1st 3y ago
  > By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
- yencabulator 3y ago
  > 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner. They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.
  - erik_seaberg 3y ago
pjmlp 3y ago
Except many of those systems are implemented on top of type 1 hypervisors, running either containers or minimal runtimes, and if there is any POSIX related APIs, it is only the minimal surface as required by ISO C and ISO C++ standards. POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.
pjc50 3y ago
As a much more low-level developer, I think they have a point, but not that the interface should be exactly S3. After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure. So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications. (not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance) We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful. > The sort of cloud storage systems described all run on top of POSIX APIs Is that a benefit or a cost?
- PaulDavisThe1st 3y ago
  I'm not disputing the idea that sometimes (much) higher level abstractions in an API are a good idea. I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.
delta_p_delta_x 3y ago
I don't quite see how web development is relevant here; I think you've missed the point. Just because it's 'object storage' doesn't mean developers are reaching for React, HTML, CSS, JavaScript, etc. Object storage is a means to put huge volumes of data elsewhere besides your own hard disk/solid state drives, or a NAS running in the basement/attic; that's all. They are a means to access that data at extremely large scale, and can work with native or web apps. More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises. There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns. Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions. [1]: https://www.techradar.com/news/microsoft-flight-simulator-2020-finally-realizes-the-true-next-generation-potential-of-azure-cloud-computing-in-pc-gaming https://www.techradar.com/news/microsoft-flight-simulator-20... Disclaimer: I am an employee at cunoFS.
- PaulDavisThe1st 3y ago
  > There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different. "I want object storage APIs for local storage" <= completely reasonable "I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable
ChrisRR 3y ago
If anything web dev is way worse than the POSIX standard as everything changes every 6-12 months. Hell, why do so many static pages even need javascript
- pjmlp 3y ago
  Because obviously we need to generate them from Next.js. /s
kazinator 3y ago
It was produced by someone thinking, "why don't I ask AI what is the difference between cloud storage APIs and POSIX".
OJFord 3y ago
It reads to me more like the title is 'flamebait' if you like, and you've been baited by it; what the article actually seems, to me, to be saying is 'POSIX good, but many things use S3-like object storage, we need a POSIX-like standard there, by the way we make cunoFS a POSIX-compat layer for "S3-compatible"'. Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.
PH95VuimJjqBqy 3y ago
fuse is a thing and it suffers from the exact same problem. You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance. And yet, it's still a very useful thing to do. The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful. Hell, the extistence of terraform should make that loud and clear.
- marcosdumay 3y ago
  Yet, the largest issues with networked filesystems is that POSIX forbids canceling file accesses, the lock semantics is really difficult, and cache is hard. POSIX is absolutely outdated, but this article isn't helpful on understanding why or how. Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.

VyseofArcadia 3y ago

Thank you for phrasing this so much more politely than I would have. I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here. Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".

jeroenhd 3y ago

"POSIX file system for the web" is kind of the whole deal for the company that wrote this. They make something to mount object storage as a filesystem. This isn't a random programmer philosophising about POSIX, it's an ad for a product.

cuno 3y ago
Hi, author here. Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
- rewmie 3y ago
  > (...) leverage the lower cost and higher throughput of object storage like S3 (...) What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
  - mcv 3y ago
    Not to mention that apparently we're supposed to compare that to storing on your local harddisk. Isn't local storage always far higher throughput than accessing something across the internet? Being able to store and access stuff on the internet as if it's local is a great idea, but the way they sell it is unconvincing and sounds like they don't really understand what they're talking about.
    - cuno 3y ago
      Hi author here, sorry I missed this post. The performance benchmarks and cost comparisons are for comparing S3 vs EBS (ext4 formatted), EFS, FSx Lustre and others within the same datacenter (i.e. LAN use case rather than WAN use case). That means if you have an EC2 instance running in, say AWS Ohio, and are comparing those storage options also within AWS Ohio, then cunoFS is both cheaper and higher throughput than those other options. It's a different story over WAN. In that case, your own local NVMe storage is going to be cheaper and generally faster that remote storage over a WAN. But that local NVMe storage (on say your solo laptop) isn't going to have anywhere near the Enterprise-grade redundancy, availability and scalability that AWS S3/Azure Blob/Storj/Wasabi/etc has.
pdntspa 3y ago
It smells like a bunch of FUD to me Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone
- psd1 3y ago
  FUD is common both ways. Your comment could be interpreted as FUD against anything new. I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)
  - pdntspa 3y ago
    Yeah, I use all of those. But what I can't really use is some low-rent sales copy trying to persuade me to abandon FOSS tooling in favor of yet another rent-seeking cloud solution. New != Good
    - psd1 3y ago
      Agreed, with only one slight quibble: content can be good even when the purpose is sales.

jasode 3y ago

>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...] The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried). Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user. So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX That's why the blog post has the following links to POSIX-on-top-of-S3-objects : https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse https://github.com/kahing/goofys https://github.com/kahing/goofys https://www.cuno.io/ https://www.cuno.io/

cuno 3y ago
Hi, author here. Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
- jonhohle 3y ago
  > in fact, in most cases, there is no filesystem at all Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)? There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).

calvinmorrison 3y ago

APIs keep getting conflate for REST APIs. As far as I understand it, the stdlib is an api

nvm0n2 3y ago

I don't know. Consider: 1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems). 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner. S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions. But ultimately, they're both storing files. The differences are fairly minor.

PaulDavisThe1st 3y ago
> By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
yencabulator 3y ago
> 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner. They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.
- erik_seaberg 3y ago

pjmlp 3y ago

Except many of those systems are implemented on top of type 1 hypervisors, running either containers or minimal runtimes, and if there is any POSIX related APIs, it is only the minimal surface as required by ISO C and ISO C++ standards. POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.

pjc50 3y ago

As a much more low-level developer, I think they have a point, but not that the interface should be exactly S3. After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure. So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications. (not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance) We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful. > The sort of cloud storage systems described all run on top of POSIX APIs Is that a benefit or a cost?

PaulDavisThe1st 3y ago
I'm not disputing the idea that sometimes (much) higher level abstractions in an API are a good idea. I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.

delta_p_delta_x 3y ago

I don't quite see how web development is relevant here; I think you've missed the point. Just because it's 'object storage' doesn't mean developers are reaching for React, HTML, CSS, JavaScript, etc. Object storage is a means to put huge volumes of data elsewhere besides your own hard disk/solid state drives, or a NAS running in the basement/attic; that's all. They are a means to access that data at extremely large scale, and can work with native or web apps. More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises. There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns. Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions. [1]: https://www.techradar.com/news/microsoft-flight-simulator-2020-finally-realizes-the-true-next-generation-potential-of-azure-cloud-computing-in-pc-gaming https://www.techradar.com/news/microsoft-flight-simulator-20... Disclaimer: I am an employee at cunoFS.

PaulDavisThe1st 3y ago
> There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different. "I want object storage APIs for local storage" <= completely reasonable "I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable

ChrisRR 3y ago

If anything web dev is way worse than the POSIX standard as everything changes every 6-12 months. Hell, why do so many static pages even need javascript

pjmlp 3y ago
Because obviously we need to generate them from Next.js. /s

kazinator 3y ago

It was produced by someone thinking, "why don't I ask AI what is the difference between cloud storage APIs and POSIX".

OJFord 3y ago

It reads to me more like the title is 'flamebait' if you like, and you've been baited by it; what the article actually seems, to me, to be saying is 'POSIX good, but many things use S3-like object storage, we need a POSIX-like standard there, by the way we make cunoFS a POSIX-compat layer for "S3-compatible"'. Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.

PH95VuimJjqBqy 3y ago

fuse is a thing and it suffers from the exact same problem. You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance. And yet, it's still a very useful thing to do. The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful. Hell, the extistence of terraform should make that loud and clear.

marcosdumay 3y ago
Yet, the largest issues with networked filesystems is that POSIX forbids canceling file accesses, the lock semantics is really difficult, and cache is hard. POSIX is absolutely outdated, but this article isn't helpful on understanding why or how. Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.