5 ms·

I think the next generation file system has CoW, are based on erasure codes and are internet distributed. For example your home NAS have InternetFS, now you can

by acd 13y ago

I think the next generation file system has CoW, are based on erasure codes and are internet distributed. For example your home NAS have InternetFS, now you can reach all your files where ever you are, all files in the home directory are always in sync. Failure of a hard drive does not matter. If your house burns down, your friends or someone else on the internet has the pieces you need to reconstruct your data. The next gen filesystem also have built in versioning via the CoW so you can always revert to an earlier version of an file. InternetFS(Encrypted, distributed, always in sync, cheap snapshots, p2p) Want to share Photos or movies with your friends and family? No problem just right click on the file, select the friend from a list, they see the file and they can view it on their computer. This next gen file system will be incredibly easy to use, cross platform. As simple to use as Facebook, Skype, Email.

rakoo 13y ago
Sound to me like Tahoe-LAFS [0] on global scale. Shall we build this ? https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs
- xiaomai 13y ago
  I was under the impression that Tahoe-LAFS doesn't work offline. Is that not the case?
  - rakoo 13y ago
    Indeed, but Tahoe-LAFS is not a replacement for your filesystem, rather a virtual place to backup and share files. Common use would be to have all files local, and a copy on your Tahoe-LAFS swarm.
- foobarqux 13y ago
  You need to run special gateway servers, I don't think it uses transparent encryption and there isn't a global namespace (for sharing publicly or with non-static groups).
  - rakoo 13y ago
    First, more reading: https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/ab... > You need to run special gateway servers You don't _need_ to, each peer can run its own gateway on localhost. Gateway servers are useful if you're on a computer where you can't install anything. > I don't think it uses transparent encryption I'm not sure what you mean by that, but if it means you need to manually encrypt things before storing it in tahoe-lafs, you're wrong. The whole point of Tahoe-LAFS is to push your files to it, and it will automatically encrypt, erasure-encode and distribute it to all nodes in the swarm you're in. All you need to care about are URIs returned by Tahoe-LAFS, and which contain all you need to use files (no need for extra password or keys, in fact they're contained in the URIs) > there isn't a global namespace Tahoe-LAFS swarms are defined by their introducers, the central piece that ties all storage nodes and potential clients together, kind of like a bittorrent tracker. There are no namespaces because swarms are not globally defined (although a node can take part in multiple swarms, this is completely invisible from Tahoe-LAFS users. Exactly like bittorrent). My proposition was to build a world-level swarm, where anybody could register and participate in the swarm. If you wonder about the reliability of a central introducer, there is work in the pipes for creating decentralized introducers.
xiaomai 13y ago
Hey! It sounds like you're describing ori [1]. Be sure to read the paper [2] because the website is a bit sparse on info. It's still young but the syncing, cheap snapshots, auto backups, etc. have been really nice. (it's only encrypted over transport though, I wish it were encrypted locally too). 1: http://ori.scs.stanford.edu/ http://ori.scs.stanford.edu/ 2: http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn=1 http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn...
- acd 13y ago
  Thanks for the ori link! :)
- foobarqux 13y ago
  Great project. Needs encryption, access controls and some kind of public namespace for publishing.
foobarqux 13y ago
I think this is obvious and also painful that no one is working on such fundamental internet infrastructure.
- wmf 13y ago
  Reliable storage is hard (thanks ChuckMcM and notacoward for educating us), distributed systems are hard, security is hard, so secure P2P erasure-coded storage is like hard^N. Would you also like it open source so it can't be monetized? Space Monkey looks like they're making good progress but I have some doubts about its scalability.
  - foobarqux 13y ago
    Yeah it's hard but fundamentally important. I think it's really difficult for such a system to be as valuable as closed source in terms of audit-ability (encryption is worthless if product is backdoored) and application development. I don't have the answer as to what would incentivize anyone to create such a system but it wouldn't be the first piece of important open source infrastructure that has been created. Space Monkey looks really interesting but I agree that this isn't something that is going to perform well running on residential internet connections. It also doesn't look like there is anything like non-explicit sharing (i.e. public files) which I think is an essential part of a distributed data store because it would be a boon for small-time publishing (no more web servers!).
Dylan16807 13y ago
So freenet but less annoying to use?
simias 13y ago
I feel like we won't see anything like that become mainstream until IPv6 becomes the norm. I would love the net to be more decentralized but everyone living behind NATs means we need central servers to connect the users between them.
nly 13y ago
My SSD gives me ~ 900 Mbps, to several Gbps, of random write performance. My ISP gives me 2 Mbps of upload bandwidth and a data cap.
- visarga 13y ago
  Differential backup. Checksumming and deduplication. If the file is from the base distribution of the OS, then chances are it is already in the cloud, so no need to back it up again.
  - dublinben 13y ago
    Who is backing up their OS? I thought most people were only backing up their unique data, which by definition wouldn't exist in 'the cloud' already.
    - ssafejava 13y ago
      Lots of people do - some backup providers charge tiered amounts or 'unlimited' data, and the convenience of getting your OS back exactly the way it was is absolutely worth using an extra 10-20GB of storage.

by acd 13y ago

rakoo 13y ago

Sound to me like Tahoe-LAFS [0] on global scale. Shall we build this ? https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs

xiaomai 13y ago
I was under the impression that Tahoe-LAFS doesn't work offline. Is that not the case?
- rakoo 13y ago
  Indeed, but Tahoe-LAFS is not a replacement for your filesystem, rather a virtual place to backup and share files. Common use would be to have all files local, and a copy on your Tahoe-LAFS swarm.
foobarqux 13y ago
You need to run special gateway servers, I don't think it uses transparent encryption and there isn't a global namespace (for sharing publicly or with non-static groups).
- rakoo 13y ago
  First, more reading: https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/ab... > You need to run special gateway servers You don't _need_ to, each peer can run its own gateway on localhost. Gateway servers are useful if you're on a computer where you can't install anything. > I don't think it uses transparent encryption I'm not sure what you mean by that, but if it means you need to manually encrypt things before storing it in tahoe-lafs, you're wrong. The whole point of Tahoe-LAFS is to push your files to it, and it will automatically encrypt, erasure-encode and distribute it to all nodes in the swarm you're in. All you need to care about are URIs returned by Tahoe-LAFS, and which contain all you need to use files (no need for extra password or keys, in fact they're contained in the URIs) > there isn't a global namespace Tahoe-LAFS swarms are defined by their introducers, the central piece that ties all storage nodes and potential clients together, kind of like a bittorrent tracker. There are no namespaces because swarms are not globally defined (although a node can take part in multiple swarms, this is completely invisible from Tahoe-LAFS users. Exactly like bittorrent). My proposition was to build a world-level swarm, where anybody could register and participate in the swarm. If you wonder about the reliability of a central introducer, there is work in the pipes for creating decentralized introducers.

xiaomai 13y ago

Hey! It sounds like you're describing ori [1]. Be sure to read the paper [2] because the website is a bit sparse on info. It's still young but the syncing, cheap snapshots, auto backups, etc. have been really nice. (it's only encrypted over transport though, I wish it were encrypted locally too). 1: http://ori.scs.stanford.edu/ http://ori.scs.stanford.edu/ 2: http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn=1 http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn...

acd 13y ago
Thanks for the ori link! :)
foobarqux 13y ago
Great project. Needs encryption, access controls and some kind of public namespace for publishing.

foobarqux 13y ago

I think this is obvious and also painful that no one is working on such fundamental internet infrastructure.

wmf 13y ago
Reliable storage is hard (thanks ChuckMcM and notacoward for educating us), distributed systems are hard, security is hard, so secure P2P erasure-coded storage is like hard^N. Would you also like it open source so it can't be monetized? Space Monkey looks like they're making good progress but I have some doubts about its scalability.
- foobarqux 13y ago
  Yeah it's hard but fundamentally important. I think it's really difficult for such a system to be as valuable as closed source in terms of audit-ability (encryption is worthless if product is backdoored) and application development. I don't have the answer as to what would incentivize anyone to create such a system but it wouldn't be the first piece of important open source infrastructure that has been created. Space Monkey looks really interesting but I agree that this isn't something that is going to perform well running on residential internet connections. It also doesn't look like there is anything like non-explicit sharing (i.e. public files) which I think is an essential part of a distributed data store because it would be a boon for small-time publishing (no more web servers!).

Dylan16807 13y ago

So freenet but less annoying to use?

simias 13y ago

I feel like we won't see anything like that become mainstream until IPv6 becomes the norm. I would love the net to be more decentralized but everyone living behind NATs means we need central servers to connect the users between them.

nly 13y ago

My SSD gives me ~ 900 Mbps, to several Gbps, of random write performance. My ISP gives me 2 Mbps of upload bandwidth and a data cap.

visarga 13y ago
Differential backup. Checksumming and deduplication. If the file is from the base distribution of the OS, then chances are it is already in the cloud, so no need to back it up again.
- dublinben 13y ago
  Who is backing up their OS? I thought most people were only backing up their unique data, which by definition wouldn't exist in 'the cloud' already.
  - ssafejava 13y ago
    Lots of people do - some backup providers charge tiered amounts or 'unlimited' data, and the convenience of getting your OS back exactly the way it was is absolutely worth using an extra 10-20GB of storage.