3 ms·

Hi, author here. Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. wit

by cuno 3y ago

Hi, author here. Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).

jonhohle 3y ago
> in fact, in most cases, there is no filesystem at all Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)? There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
- cuno 3y ago
  Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.
- IcePic 3y ago
  CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it). But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.
  - jonhohle 3y ago
    Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.
    - tryauuum 3y ago
      https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf my google-fu found this it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs
    - IcePic 3y ago
      If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.
- LightFog 3y ago
  It’s not uncommon for high performance object stores to bypass POSIX FS semantics and provide kernel patches etc - eg lustre do this.
chasil 3y ago
One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash. I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do? Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox. Could the POSIX standard shell move that far? Or is there absolutely no hope? https://frippery.org/busybox/index.html https://frippery.org/busybox/index.html
- ilyt 3y ago
  > One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash. Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that https://lwn.net/Articles/343924/ https://lwn.net/Articles/343924/ And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel. At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
  - chasil 3y ago
    This is very much not true. "A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K. "ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach." Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays. This is why POSIX Powershell could never exist. https://news.slashdot.org/story/01/02/06/2030205/david-korn-tells-all https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
    - chasil 3y ago
      Interesting how expressed frustration with POSIX.2 is received.
nickik 3y ago
ZFS new object store backend give you that interface I think.

by cuno 3y ago

jonhohle 3y ago

> in fact, in most cases, there is no filesystem at all Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)? There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).

cuno 3y ago
Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.
IcePic 3y ago
CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it). But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.
- jonhohle 3y ago
  Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.
  - tryauuum 3y ago
    https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf my google-fu found this it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs
  - IcePic 3y ago
    If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.
LightFog 3y ago
It’s not uncommon for high performance object stores to bypass POSIX FS semantics and provide kernel patches etc - eg lustre do this.

chasil 3y ago

One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash. I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do? Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox. Could the POSIX standard shell move that far? Or is there absolutely no hope? https://frippery.org/busybox/index.html https://frippery.org/busybox/index.html

ilyt 3y ago
> One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash. Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that https://lwn.net/Articles/343924/ https://lwn.net/Articles/343924/ And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel. At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
- chasil 3y ago
  This is very much not true. "A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K. "ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach." Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays. This is why POSIX Powershell could never exist. https://news.slashdot.org/story/01/02/06/2030205/david-korn-tells-all https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
  - chasil 3y ago
    Interesting how expressed frustration with POSIX.2 is received.

nickik 3y ago

ZFS new object store backend give you that interface I think.