6 ms·

>What we need is not [...] cloud storage. We need [...] a decent distributed filesystem, Distributed files to where exactly? You need to be more concrete abou

by jasode 4y ago

>What we need is not [...] cloud storage. We need [...] a decent distributed filesystem, Distributed files to where exactly? You need to be more concrete about the remote location of non-local data that normal people can use. Ok, so you want "distributed filesystem" to not mean "cloud storage" ... So is it p2p? Something else? In other words, we want Windows, macOS, Linux, iPhone, Android, etc operating systems... to have a file system that all points to the same "distributed filesystem" and see the same files -- and for other collaborators to see those files. But we don't want those os configurations to point to DropBox / MS OneDrive / Google Drive / Backblaze, etc. So, we need to be concrete on the alternative common remote location that those file system APIs would point to. What would the topology of that solution look like?

oriolid 4y ago
This is the problem that needs to be solved. Cloud storage and p2p are solutions looking for a problem, but it would be nice to let them distract us too much.
jagged-chisel 4y ago
> Distributed files to where exactly? My desktop computer, my laptop computer, my tablet computer, my pocket computer. Whether that’s cloud or p2p doesn’t matter to me, the user. I should be able to start working on a spreadsheet or presentation on one and, without the ceremony of “save to a shared location, close the app, switch devices, open the app … now where’s that file again?” switch to another and continue editing. First we need to specify a distributed fs. THEN we can decide the “to where” bit.
- generalizations 4y ago
  That sounds like syncthing. Dunno about phones and tablets, but I have that functionality among my computers.
  - josteink 4y ago
    I use syncthing. It's awesome for computers. Coming from Dropbox, then Nextcloud, I find it solves all my needs much, much better, at least on well-supported platforms. I love how I can decide what to sync where, and even create my own topology of sync-devices if I like. That may sound like crazy complex stuff and over-engineering and what not, but it was a solution I landed on organically, just through normal use. That said, it's not entirely smooth on iOS and you sometimes needs to manually launch the (third party) app to force a sync after changing some files.
    - deleted 4y ago
      [deleted]
  - pluijzer 4y ago
    I have no affiliation but want to second this. If you want to keep your filesystem in sync across many devices Syncthing fully enables this. It is a use case where you would expect a payed service would be easier or more reliable but with Syncthing it is the exact opposite. Just install it on your devices and select the folders you want to keep in sync ... done. I had never had any problems with it something I cannot say about Dropbox which can be terribly slow, hogs my PC and resulted in lost files on some occasion.
    - Rygian 4y ago
      My filesystem consists of [checks du -h . | tail -1] 189 GB. I don't think Syncthing (which I love) can cram 189 GB on my 64 GB phone. Yet I expect to have access to my filesystem from my phone. Synthing is a nice "pump-hose system" between reservoirs of data. What I was arguing above is to stop having separate reservoirs of data to begin with.
      - jcelerier 4y ago
        I use seafile (similar to syncthing) and it allows to browse your data libraries without requiring a local copy for these cases
- franga2000 4y ago
  The "to where" bit is really important when specifying the fs. If it includes at least one high-bandwidth high-storage high-uptime device (like a server), the requirements and capabilities change drastically compared to if it's composed of a bunch of battery-powered portable devices on limited data plans.
- resizeitplz 4y ago
  Funny you use a spreadsheet as an example. That's been the default for Excel for years. Save to SharePoint/Teams/OneDrive (whatever MS is calling it these days, it's all the same backend) is the default option - and multi-user live editing (or one user in multiple sessions) just works.
- lupire 4y ago
  Ray Ozzie, is that you?! https://news.ycombinator.com/item?id=5661920 https://news.ycombinator.com/item?id=5661920 https://www.joelonsoftware.com/2008/05/01/architecture-astronauts-take-over/ https://www.joelonsoftware.com/2008/05/01/architecture-astro... Come back to us, Microsoft Briefcase!
Rygian 4y ago
To where exactly: my devices. And if I don't want to buy my own devices, then to a cloud service that offers opaque storage of binary blobs with an API that my filesystem can abstract for me. So the topology is a mesh network of my devices, and perhaps optionally a few defined remote endpoints that the opaque blob storage service provides me, and that I enter as part of the config of my filesystem.
- lupire 4y ago
  That sounds like Cloud Storage to me (Dropbox, Google Drive Backup/Restore/Sync/whatever it's called this year).
  - Rygian 4y ago
    Cloud storage is the opposite of distributed. With cloud storage, you must have one single fixed central location (often a third party) that contains the real data, many satellite locations with a partial replica of the data, and hit-or-miss mechanisms to notice changes in replicas and propagate them to the central location. If the central location is down there is no more synchronization. If the central location is not yours, they can shut you down anytime. A distributed filesystem does away with the need for a fixed central storage by storing data across all locations with a configurable level of replication. A strong, consistent cascading of changes (eg a crdt semantic) brings all replicas in sync whenever connectivity allows. No third parties need to be involved, no single device is a point of failure.

by jasode 4y ago

oriolid 4y ago

This is the problem that needs to be solved. Cloud storage and p2p are solutions looking for a problem, but it would be nice to let them distract us too much.

jagged-chisel 4y ago

> Distributed files to where exactly? My desktop computer, my laptop computer, my tablet computer, my pocket computer. Whether that’s cloud or p2p doesn’t matter to me, the user. I should be able to start working on a spreadsheet or presentation on one and, without the ceremony of “save to a shared location, close the app, switch devices, open the app … now where’s that file again?” switch to another and continue editing. First we need to specify a distributed fs. THEN we can decide the “to where” bit.

generalizations 4y ago
That sounds like syncthing. Dunno about phones and tablets, but I have that functionality among my computers.
- josteink 4y ago
  I use syncthing. It's awesome for computers. Coming from Dropbox, then Nextcloud, I find it solves all my needs much, much better, at least on well-supported platforms. I love how I can decide what to sync where, and even create my own topology of sync-devices if I like. That may sound like crazy complex stuff and over-engineering and what not, but it was a solution I landed on organically, just through normal use. That said, it's not entirely smooth on iOS and you sometimes needs to manually launch the (third party) app to force a sync after changing some files.
  - deleted 4y ago
    [deleted]
- pluijzer 4y ago
  I have no affiliation but want to second this. If you want to keep your filesystem in sync across many devices Syncthing fully enables this. It is a use case where you would expect a payed service would be easier or more reliable but with Syncthing it is the exact opposite. Just install it on your devices and select the folders you want to keep in sync ... done. I had never had any problems with it something I cannot say about Dropbox which can be terribly slow, hogs my PC and resulted in lost files on some occasion.
  - Rygian 4y ago
    My filesystem consists of [checks du -h . | tail -1] 189 GB. I don't think Syncthing (which I love) can cram 189 GB on my 64 GB phone. Yet I expect to have access to my filesystem from my phone. Synthing is a nice "pump-hose system" between reservoirs of data. What I was arguing above is to stop having separate reservoirs of data to begin with.
    - jcelerier 4y ago
      I use seafile (similar to syncthing) and it allows to browse your data libraries without requiring a local copy for these cases
franga2000 4y ago
The "to where" bit is really important when specifying the fs. If it includes at least one high-bandwidth high-storage high-uptime device (like a server), the requirements and capabilities change drastically compared to if it's composed of a bunch of battery-powered portable devices on limited data plans.
resizeitplz 4y ago
Funny you use a spreadsheet as an example. That's been the default for Excel for years. Save to SharePoint/Teams/OneDrive (whatever MS is calling it these days, it's all the same backend) is the default option - and multi-user live editing (or one user in multiple sessions) just works.
lupire 4y ago
Ray Ozzie, is that you?! https://news.ycombinator.com/item?id=5661920 https://news.ycombinator.com/item?id=5661920 https://www.joelonsoftware.com/2008/05/01/architecture-astronauts-take-over/ https://www.joelonsoftware.com/2008/05/01/architecture-astro... Come back to us, Microsoft Briefcase!

Rygian 4y ago

To where exactly: my devices. And if I don't want to buy my own devices, then to a cloud service that offers opaque storage of binary blobs with an API that my filesystem can abstract for me. So the topology is a mesh network of my devices, and perhaps optionally a few defined remote endpoints that the opaque blob storage service provides me, and that I enter as part of the config of my filesystem.

lupire 4y ago
That sounds like Cloud Storage to me (Dropbox, Google Drive Backup/Restore/Sync/whatever it's called this year).
- Rygian 4y ago
  Cloud storage is the opposite of distributed. With cloud storage, you must have one single fixed central location (often a third party) that contains the real data, many satellite locations with a partial replica of the data, and hit-or-miss mechanisms to notice changes in replicas and propagate them to the central location. If the central location is down there is no more synchronization. If the central location is not yours, they can shut you down anytime. A distributed filesystem does away with the need for a fixed central storage by storing data across all locations with a configurable level of replication. A strong, consistent cascading of changes (eg a crdt semantic) brings all replicas in sync whenever connectivity allows. No third parties need to be involved, no single device is a point of failure.