3 ms·

The manifesto's idea that files should have a unique address which any machine can access reminds me of Brian Hauer's rant, which argues that each application s

The manifesto's idea that files should have a unique address which any machine can access reminds me of Brian Hauer's rant, which argues that each application should have a single instance with a unique address (for a given user) that any machine can access (http://tiamat.tsotech.com/pao http://tiamat.tsotech.com/pao). Put the two together and a person's entire digital life would seamlessly follow them between machines. I like the proposal of making caching a central design element to work around today's bandwidth limitations. I work with large-ish (a few TB) scientific datasets, and it isn't pleasant to have to choose between (a) storing everything on network storage and suffering slow IO or (b) storing everything locally on every workstation and suffering the need to synchronize data.

scoot 9y ago
" it isn't pleasant to have to choose between (a) storing everything on network storage and suffering slow IO or (b) storing everything locally on every workstation and suffering the need to synchronize data." That's a solved problem, which depending on the workload could involve a compute farm with a clustered or distributed file system, a copy data management solution, or cachefs, for example. The are also solutions for shared storage between containers across multiple nodes.
- jpeloquin 9y ago
  > That's a solved problem ... compute farm with a clustered or distributed file system, a copy data management solution, or cachefs, for example. Thank you for suggesting some potential solutions to my data management problems. From googling them, I get the impression that you are thinking primarily of enterprise scale data management (i.e., organizations with server farms and an IT department), whereas I'm primarily thinking of organizations with < 30 employees (who mostly use desktop software) and a single file server. My particular situation is an academic lab. However, I think these solutions can can still work with a little adaptation: Copy data management seems to be the use of block level data deduplication or virtual disks on a server in order to decrease the disk utilization per VM or per container. I'm not completely sure; I found mostly marketing documents, and there's no wikipedia entry. This, as well as clustered/distributed file systems, would apply if we turned the file server into a VM server and had each employee work on a VM via remote desktop. In principle, this could work, and would let temporary employees (e.g., summer students and visiting scholars) get started quickly with a standard OS environment. I will experiment with this when our last batch of desktop PCs hit end of life. CacheFS looks useful if we start using NFS to connect to our file server instead of Samba. It looks like Windows 10 (Enterprise) supports NFS caching too (https://technet.microsoft.com/en-us/library/cc976862.aspx https://technet.microsoft.com/en-us/library/cc976862.aspx). I will try this.

scoot 9y ago

" it isn't pleasant to have to choose between (a) storing everything on network storage and suffering slow IO or (b) storing everything locally on every workstation and suffering the need to synchronize data." That's a solved problem, which depending on the workload could involve a compute farm with a clustered or distributed file system, a copy data management solution, or cachefs, for example. The are also solutions for shared storage between containers across multiple nodes.

jpeloquin 9y ago
> That's a solved problem ... compute farm with a clustered or distributed file system, a copy data management solution, or cachefs, for example. Thank you for suggesting some potential solutions to my data management problems. From googling them, I get the impression that you are thinking primarily of enterprise scale data management (i.e., organizations with server farms and an IT department), whereas I'm primarily thinking of organizations with < 30 employees (who mostly use desktop software) and a single file server. My particular situation is an academic lab. However, I think these solutions can can still work with a little adaptation: Copy data management seems to be the use of block level data deduplication or virtual disks on a server in order to decrease the disk utilization per VM or per container. I'm not completely sure; I found mostly marketing documents, and there's no wikipedia entry. This, as well as clustered/distributed file systems, would apply if we turned the file server into a VM server and had each employee work on a VM via remote desktop. In principle, this could work, and would let temporary employees (e.g., summer students and visiting scholars) get started quickly with a standard OS environment. I will experiment with this when our last batch of desktop PCs hit end of life. CacheFS looks useful if we start using NFS to connect to our file server instead of Samba. It looks like Windows 10 (Enterprise) supports NFS caching too (https://technet.microsoft.com/en-us/library/cc976862.aspx https://technet.microsoft.com/en-us/library/cc976862.aspx). I will try this.