4 ms·

> What I learned though was that NFS was great until it wasn't. If the server hung, all work stopped. Sheds a tear for AFS (Andrew File System). We had a nice

by bsder 1y ago

> What I learned though was that NFS was great until it wasn't. If the server hung, all work stopped. Sheds a tear for AFS (Andrew File System). We had a nice, distributed file system that even had solid security and didn't fail in these silly ways--everybody ignored it.

slater 1y ago
Looks like these guys are still truckin' along? https://www.openafs.org/ https://www.openafs.org/
nine_k 1y ago
AFS implements weak consistency, which may be a bit surprising. It also seems to share objects, not block devices. Judging by its features, it seems to make most sense when there is a cluster of servers. It looks cool though, a bit more like S3 than like NFS.
- jaltman 1y ago
  The cephfs model of a file system logically constructed from an object store closely mirrors the AFS architecture. The AFS fileserver is horribly misnamed. Whereas AFS 1.0 fileserver exported the contents of local filesystems much as NFS and CIFS do, AFS 2.x/3.x/OpenAFS/AuriStorFS fileservers export objects (aka vnodes) which are stored in an object store. Each AFS vice partition stored zero or more object stores each consisting of the objects belonging to a single volume group. A volume group consists of one or more of the RWVOL, ROVOL and/or BACKVOL instances. The AFS consistency model is fairly strong. Each client (aka cache manager) is only permitted to access the data/metadata of a vnode if it has been issued a callback promise from the AFS fileserver. File lock transitions, metdata modifications, and data modifications as well as volume transactions cause the fileserver to break the promise. At which point the client is required to fetch updated status information before it can decide it is safe to reuse the locally cached data. Unlike optimistic locking models, the AFS model permits cached data to be validated after an extended period of time by requesting up to date metadata and a new callback promise. An AFS fileserver will not permit a client to perform a state changing operation as long as there exist broken callback promises which have yet to be successfully delivered to the client.
cherrycherry98 1y ago
Morgan Stanley was a heavy user of AFS for deploying software and might still be for all I know. "Most Production Applications run from AFS" "Most UNIX hosts are dataless AFS clients" https://web.archive.org/web/20170709042700/http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf https://web.archive.org/web/20170709042700/http://www-conf.s...
- hinkley 1y ago
  NCSA also used it for some data archival and I believe for hosting the website files. I looked up at one point whatever happened to AFS and it turns out that it has some Amdahl’s Law glass ceiling that ultimately limits the aggregate bandwidth to something around 1 GBps, which was fine when it was young but not fine when 100Mb Ethernet was ubiquitous and gigabit was obtainable with deep enough pockets. If adding more hardware can’t make the filesystem faster you’re dead. I don’t know if or how openAFS has avoided these issues.
  - dotwaffle 1y ago
    I know quite a few AFS systems that moved to AuriStor's YFS: https://www.auristor.com/openafs/migrate-to-auristor/auristor-comparison https://www.auristor.com/openafs/migrate-to-auristor/auristo... As I understand it, it mitigated many of those issues, but is still very "90s" in operation. I've been flirting with the idea of writing a replacement for years, about time I had a go at it!
    - hinkley 1y ago
      I may be confusing two systems but I believe that AFS system was also encompassed the first iteration of “AWS Glacier” I encountered in the wild. A big storage that required queuing a job to a tape array or pinging an undergrad to manually load something for retrieval.
  - jaltman 1y ago
    The Amdahl's Law limitations are specific to the implementation and not at all tied to the protocols. The 1990 AFS 3.0 server design was built upon a cooperative threading system ("Light Weight Processes") designed by James Gosling as part of the Andrew Project. Cooperative processing influences the design of the locking model since there isn't any simultaneous between tasks. When the AFS fileserver was converted to pthreads for AFS 3.5, the global state of each library was protected by wrapping it with a global mutex. Each mutex was acquired when entering the library and dropped when exiting it. To complete any fileserver RPC required acquisition of at least six or seven global mutexes depending upon the type of vnode being be accessed. In practice, the global mutexes restricted the fileserver process to 1.7 cores regardless of how many cores were present in the system. AuriStor's RX and UBIK protocol and implementation improvements would be worthless if the application services couldn't scale. To accomplish this required converting each subsystem so it could operate with minimal lock contention. This 2023 presentation by Simon Wilkinson describes the improvements that were made to AuriStor's RX implementation up to that point. https://www.auristor.com/downloads/auristor-rx-hare-and-the-tortoise-2023.pdf https://www.auristor.com/downloads/auristor-rx-hare-and-the-... The RX tortoise is catching up with the TCP hare. Connecting to [10.0.2.15]:2345 RECV: threads 1, times 1, bytes 2000000000: 881 msec [18.15 Gbit/s]
tempay 1y ago
I'm still in a place where AFS is chugging along for many years after it's much exaggarated demise. It never ceases to amaze me how well it does what it does and how well it handles being misused.
fuzztester 1y ago
Interesting. I think I had casually come across the term AFS (and its full form) back in the day, but never look into it deeply. Why did everybody ignore it, do you know?
- jaltman 1y ago
  Not everyone ignored it but unlike nfs it didn't come in the box with the operating system, and you had to pay for it. In addition, AFS provided strong cryptographic authentication and wire privacy which meant that it couldn't be licensed in many countries because the U.S. government did not grant appropriate export licenses. I often wonder how the world would be different if AFS 3.0 could have been freely distributed world wide in 1989 precluding the need for HTTP to be developed at CERN.
- bsder 1y ago
  There were a few technical obstacles which other people mentioned, but I think timing was biggest issue (remember--AFS dates to something like 1983-ish). 1) AFS, IIRC, required more than one machine in its original configuration. That meant hardware and sysadmins which were expensive--until, suddenly they weren't. 2) Disk, memory and bandwidth were scarce--and then they weren't. AFS made a bunch of solid architectural decisions and then wasted a bunch of time backing some of them down in deference to the hardware of the day and then all that work was wasted when Moore's Law overran everything, anyhow. 3) Everybody was super happy to be running everything locally to escape the tyranny of the "Mainframe Operator" (meaning no NFS or AFS or the like)--until they weren't. Once enough non-technical people appeared who didn't want to do system administration, like, ever, that flipped. We lost the VMS filesystem in this timeframe, too. Which was also a distributed, remote filesystem. But those x86 processors sure are cheap ... sigh.
burnt-resistor 1y ago
And heavily used by top universities for the past ~35 years.

by bsder 1y ago

slater 1y ago

Looks like these guys are still truckin' along? https://www.openafs.org/ https://www.openafs.org/

nine_k 1y ago

AFS implements weak consistency, which may be a bit surprising. It also seems to share objects, not block devices. Judging by its features, it seems to make most sense when there is a cluster of servers. It looks cool though, a bit more like S3 than like NFS.

jaltman 1y ago
The cephfs model of a file system logically constructed from an object store closely mirrors the AFS architecture. The AFS fileserver is horribly misnamed. Whereas AFS 1.0 fileserver exported the contents of local filesystems much as NFS and CIFS do, AFS 2.x/3.x/OpenAFS/AuriStorFS fileservers export objects (aka vnodes) which are stored in an object store. Each AFS vice partition stored zero or more object stores each consisting of the objects belonging to a single volume group. A volume group consists of one or more of the RWVOL, ROVOL and/or BACKVOL instances. The AFS consistency model is fairly strong. Each client (aka cache manager) is only permitted to access the data/metadata of a vnode if it has been issued a callback promise from the AFS fileserver. File lock transitions, metdata modifications, and data modifications as well as volume transactions cause the fileserver to break the promise. At which point the client is required to fetch updated status information before it can decide it is safe to reuse the locally cached data. Unlike optimistic locking models, the AFS model permits cached data to be validated after an extended period of time by requesting up to date metadata and a new callback promise. An AFS fileserver will not permit a client to perform a state changing operation as long as there exist broken callback promises which have yet to be successfully delivered to the client.

cherrycherry98 1y ago

Morgan Stanley was a heavy user of AFS for deploying software and might still be for all I know. "Most Production Applications run from AFS" "Most UNIX hosts are dataless AFS clients" https://web.archive.org/web/20170709042700/http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf https://web.archive.org/web/20170709042700/http://www-conf.s...

hinkley 1y ago
NCSA also used it for some data archival and I believe for hosting the website files. I looked up at one point whatever happened to AFS and it turns out that it has some Amdahl’s Law glass ceiling that ultimately limits the aggregate bandwidth to something around 1 GBps, which was fine when it was young but not fine when 100Mb Ethernet was ubiquitous and gigabit was obtainable with deep enough pockets. If adding more hardware can’t make the filesystem faster you’re dead. I don’t know if or how openAFS has avoided these issues.
- dotwaffle 1y ago
  I know quite a few AFS systems that moved to AuriStor's YFS: https://www.auristor.com/openafs/migrate-to-auristor/auristor-comparison https://www.auristor.com/openafs/migrate-to-auristor/auristo... As I understand it, it mitigated many of those issues, but is still very "90s" in operation. I've been flirting with the idea of writing a replacement for years, about time I had a go at it!
  - hinkley 1y ago
    I may be confusing two systems but I believe that AFS system was also encompassed the first iteration of “AWS Glacier” I encountered in the wild. A big storage that required queuing a job to a tape array or pinging an undergrad to manually load something for retrieval.
- jaltman 1y ago
  The Amdahl's Law limitations are specific to the implementation and not at all tied to the protocols. The 1990 AFS 3.0 server design was built upon a cooperative threading system ("Light Weight Processes") designed by James Gosling as part of the Andrew Project. Cooperative processing influences the design of the locking model since there isn't any simultaneous between tasks. When the AFS fileserver was converted to pthreads for AFS 3.5, the global state of each library was protected by wrapping it with a global mutex. Each mutex was acquired when entering the library and dropped when exiting it. To complete any fileserver RPC required acquisition of at least six or seven global mutexes depending upon the type of vnode being be accessed. In practice, the global mutexes restricted the fileserver process to 1.7 cores regardless of how many cores were present in the system. AuriStor's RX and UBIK protocol and implementation improvements would be worthless if the application services couldn't scale. To accomplish this required converting each subsystem so it could operate with minimal lock contention. This 2023 presentation by Simon Wilkinson describes the improvements that were made to AuriStor's RX implementation up to that point. https://www.auristor.com/downloads/auristor-rx-hare-and-the-tortoise-2023.pdf https://www.auristor.com/downloads/auristor-rx-hare-and-the-... The RX tortoise is catching up with the TCP hare. Connecting to [10.0.2.15]:2345 RECV: threads 1, times 1, bytes 2000000000: 881 msec [18.15 Gbit/s]

tempay 1y ago

I'm still in a place where AFS is chugging along for many years after it's much exaggarated demise. It never ceases to amaze me how well it does what it does and how well it handles being misused.

fuzztester 1y ago

Interesting. I think I had casually come across the term AFS (and its full form) back in the day, but never look into it deeply. Why did everybody ignore it, do you know?

jaltman 1y ago
Not everyone ignored it but unlike nfs it didn't come in the box with the operating system, and you had to pay for it. In addition, AFS provided strong cryptographic authentication and wire privacy which meant that it couldn't be licensed in many countries because the U.S. government did not grant appropriate export licenses. I often wonder how the world would be different if AFS 3.0 could have been freely distributed world wide in 1989 precluding the need for HTTP to be developed at CERN.
bsder 1y ago
There were a few technical obstacles which other people mentioned, but I think timing was biggest issue (remember--AFS dates to something like 1983-ish). 1) AFS, IIRC, required more than one machine in its original configuration. That meant hardware and sysadmins which were expensive--until, suddenly they weren't. 2) Disk, memory and bandwidth were scarce--and then they weren't. AFS made a bunch of solid architectural decisions and then wasted a bunch of time backing some of them down in deference to the hardware of the day and then all that work was wasted when Moore's Law overran everything, anyhow. 3) Everybody was super happy to be running everything locally to escape the tyranny of the "Mainframe Operator" (meaning no NFS or AFS or the like)--until they weren't. Once enough non-technical people appeared who didn't want to do system administration, like, ever, that flipped. We lost the VMS filesystem in this timeframe, too. Which was also a distributed, remote filesystem. But those x86 processors sure are cheap ... sigh.

burnt-resistor 1y ago

And heavily used by top universities for the past ~35 years.