6 ms·

Hey everyone, I'm the author of WebFS. Happy to answer any questions.

jerry292 7y ago
I read you use IPFS. To my knowledge everything on there can be encrypted however not private. Is there a specific way you get around that.
- brendoncarroll 7y ago
  Yes. WebFS doesn't actually use any of the file/directory functionality provided by IPFS, or any encryption features. We only use the get/put block functionality. Everything is encrypted in WebFS before being posted to a Store. The data encryption keys are generated using a secret and the hash of the data being encrypted. That key is stored in the reference to that data. This continues recursively to the superblock which is not encrypted.
  - mirimir 7y ago
    Can one use Tor onions as stores?
    - sp332 7y ago
      Tor is a network layer. It doesn't really store anything.
      - mirimir 7y ago
        True. But Tor onions are servers.
        duskwuff 7y ago
        Not in any sense that's relevant here. They don't host content.
        mirimir 7y ago
        I have no clue where you're coming from. What sort of content do you say Tor onions can't host? "Tor onion" just means that a server is (ideally) only reachable as an onion URL, which is only accessible via the Tor network. There is the limitation that Tor only handles TCP. Otherwise, one can route anything over Tor. In my experience, that includes HTTP(S), FTP, Tahoe-LAFS, SSH, RDP, Mumble, OpenVPN and tinc. And others, if I spent more time remembering what I've played with.
        duskwuff 7y ago
        Okay, I think I see the source of the confusion. What you're describing is a Tor hidden service. Hidden services are separate from the Tor relay network itself, which is what I thought you were referring to as "Tor onions". Hidden services are optimized for confidentiality over performance. Using them for bulk data storage would place a lot of load on the relay network, and it's not clear what security problem this arrangement would solve.
        mirimir 7y ago
        Again, sorry. As far as I know, "hidden service" is deprecated, with "onion service" the current term. And it does tend to get shortened to "onion". But I admit that it was confusing. Because relays used to be called "onion routers". Which is also more or less deprecated, I think. The security problem is Stores being physically located and compromised, based on IP addresses found in traffic logs.
        sp332 7y ago
        I've never heard of an onion service called just "an onion" before. I don't know enough to say if that's wrong, just that I was confused.
        mirimir 7y ago
        Sorry to be confusing. It is commonly used, in some Tor communities. But yes, "onion service" is clearer.
- mirimir 7y ago
  What do you mean by "private"? I mean, if it's securely encrypted, does it matter if others can see it? And indeed, if it's online, you must assume that others can see it. My privacy concern is about IP addresses. So I'd want to use IPFS with Tor onions as stores.
  - jerry292 7y ago
    Well I fear even if the information is encrypted alone once quantum computing breaks modern AES encryption standards that’s going to be a yikes. So I’d be more comfortable with encryption as well as access controls.
    - mirimir 7y ago
      I guess. But access controls really just keep the punters out. Any serious adversary will just track down the stores. And even if they're on dedicated servers with FDE, keys can be obtained from RAM.
      - brendoncarroll 7y ago
        Just to clarify: All data is encrypted on the client, going after a server backing a Store will get you encrypted blobs. Encryption keys would not exist on the server in plaintext.
        mirimir 7y ago
        Sorry to confuse the issue. I meant the keys for the Store's FDE, not the WebFS keys, which never leave the user's machine. I was addressing jerry292's concern about access to the encrypted data in Stores.
    - brendoncarroll 7y ago
      This is a legitimate concern. WebFS is designed for the p2p storage use case. Persisting data with p2p storage means that it can live forever. All the secrets in WebFS are randomly generated and there are no user supplied (potentially weak) passwords. w.r.t. quantum computing: it is possible for WebFS to use symmetric cryptography for all remote data. Although, many Cell implementations in the near term will likely use elliptic curves or RSA.
toomim 7y ago
One issue that networked filesystems have is with mutations, especially with multiple writers. WebDAV, NFS, etc. try to address this with locking, but that doesn't allow simultaneous writes, which means it's dangerous to work offline. It's possible to solve this by using an OT or CRDT algorithm behind the scene. This is what we are building into an extension of HTTP called braid (https://braid.news https://braid.news). It could be useful for a web-backed filesystem. It automates synchronization.
- layoutIfNeeded 7y ago
  Thanks for the shameless plug, but what does this have to do with WebFS?
mirimir 7y ago
ELI5 please. Where is the data for all of those IPFS hashes stored? And what ensures that it will persist?
- brendoncarroll 7y ago
  WebFS stores data in things it calls "Stores". Stores can be anything that takes data and gives back a key to retrieve the data. Right now we have an IPFS and HTTP store built. An FTP server could also work as a store. If you or a friend run the HTTP or FTP server then it will persist the data for you. IPFS doesn't incentivize data persistence so if WebFS is working on top of IPFS it inherits that problem. You could run WebFS on top of one of the storage networks and persisting your blobs would be incentivized. WebFS is storage layer agnostic. Give it a Store, and it will give you a file system.
  - mirimir 7y ago
    OK, cool. I understand now. So if WebFS is running on a system with access to Tor SocksPorts, can Stores be onion URLs? Edit: If not, one could presumably route WebFS through OnionCat's IPv6 /48. But that only works with v2 onions, which are deprecated. However, tinc works with v3 onions. And either of those gives you UDP transport.
    - brendoncarroll 7y ago
      Yes, you could use the HTTP Store over Tor. I think you would have to configure the proxy environment variables for Go's HTTP client.
      - mirimir 7y ago
        Thanks. I'll try it, and comment on GitHub.
rkagerer 7y ago
Your page mentions files in a store are identified by a hash of their contents. Does this have any privacy implications? Can another participant in the network tell what files are in my store? (like a party providing disk space, or someone intercepting my traffic)
- brendoncarroll 7y ago
  Encryption keys are derived from a secret and the hash of the data. The secrets are set per Volume. An empty secret would be a convergent encryption strategy, maybe suitable for public data. If the secret is set then the keys will be convergent with other keys generated with the same secret. This amounts to deduplication within a Volume, and privacy. All a storage provider sees are many small encrypted blobs, so the size of large files is not leaked either.
dredmorbius 7y ago
I'm going to assume you're familiar with Plan9OS's WebFS and 9P Protcol? https://en.m.wikipedia.org/wiki/9P_(protoco) https://en.m.wikipedia.org/wiki/9P_(protoco) I've been kicking around similar ideas (nowhere near implemented) for a while: https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if_the_web_was_filesystemaccessible/ https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if...
- brendoncarroll 7y ago
  I was not aware of Plan 9's WebFS. It looks like it presents websites in the local file system. IPFS can do something very similar with its content. I have used that before. The name here comes from using web resources (referred to by url) as building blocks for a file system. I guess "Web for FS" rather than "Web as FS".
  - dredmorbius 7y ago
    The notion of remote services accessed via local filesystem dynamics is pretty well established. Among implementations: - NFS, particularly with the Solaris-originated concept of automounts over a /net mountpoint. - Various virtual filesystems. Midnight Commander ("mc") offers several of these, including archive formats (tar, cpio, afio, rpm, deb) and remote (FTP, SSH). - SMB/CFS/Samba - Various FUSE filesystems, including again ssh, ftp, and others. These generally require specifying in advance specific mountpoints. The notion of on-demand access to remote resources over protocols (e.g., http/https, or others), under filesystem dynamics, is interesting -- you can use any general tool, utility, or application for access, mediated through the filesystem by way of drivers, rather than a specific application (e.g., Web browser, FTP client, etc.) There are numerous issues. In particular, applications tend not to respond well to remote resources disappearing, changing, or failing to return from change requests -- NFS's behaviour with nonresponsive remote hosts is ... notorious. Consistency, availability, and partition resistance (CAP) are long-standing concerns, and there's no way to solve for all three. I'd add latency as another major consideration. https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem The general notion of managing and tracking changes locally, and pushing them to remote, has merits. I'm not aware of a "gitfs" ... though of course, one does exist, TIL: https://www.presslabs.com/code/gitfs/ https://www.presslabs.com/code/gitfs/ The notion of using git (or other versioning system) as a mediator for remote/local revisioned access seems to have merits. Obviously not viable for very-high-variability systems, but adequate for many occasionally-modified resources. I'm not sure if you're looking at using your WebFS itself as a publishing mechanism, though in general I think I'd recommend against doing this. For small-n peer-to-peer distribution that's probably workable, but for large-scale provisioning-and-request systems, relying on HTTP or other established transports is likely more sensible. One area I've recognised as being particularly fraught is the whole notion of security and privacy. Providing unfiltered local access to remote resources which may change arbitrarily is a great way for allowing malware onto local systems -- your transport layer should probably implement some level of security and mounts deny direct execution of content. The fact that remote content could be copied to an executable mountpoint remains, and would make numerous attacks possible. Similarly: access, update, write, and/or publishing actions all leak considerable information which could be of concern to specific users or organisations. Hash-based indexing (already addressed in this thread) being only one of several such vectors.
cbsmith 7y ago
Name is dangerously close to WebNFS.
mpweiher 7y ago
This looks really cool. > If you have ever thought "x can probably be used as a file system" ...you might also want to take a look at Storage Combinators[1]. Not quite the same problem space, but abstracting away a bit from both concrete filesystems and other storage mechanisms to get to a composable abstraction. Note: I am the primary author, and also taking a good look at WebFS for further inspiration... :-) [1] https://2019.splashcon.org/details/splash-2019-Onward-papers/7/Storage-Combinators https://2019.splashcon.org/details/splash-2019-Onward-papers...
ianopolous 7y ago
You will probably be interested in Peergos [0][1], which at it's lowest level is an encrypted global filesystem also built on IPFS and also only using the block api. [0] https://github.com/peergos/peergos https://github.com/peergos/peergos [1] https://book.peergos.org https://book.peergos.org
jerrygreenest 7y ago
I see very little docs and absolutely no fancy info like gifs or videos explaining what it's possible to do with this tool. I see some example folder, but I can't really say the difference between webfs and ipfs. (Just a little feedback, I mean no offence)

I read you use IPFS. To my knowledge everything on there can be encrypted however not private. Is there a specific way you get around that.

brendoncarroll 7y ago
Yes. WebFS doesn't actually use any of the file/directory functionality provided by IPFS, or any encryption features. We only use the get/put block functionality. Everything is encrypted in WebFS before being posted to a Store. The data encryption keys are generated using a secret and the hash of the data being encrypted. That key is stored in the reference to that data. This continues recursively to the superblock which is not encrypted.
- mirimir 7y ago
  Can one use Tor onions as stores?
  - sp332 7y ago
    Tor is a network layer. It doesn't really store anything.
    - mirimir 7y ago
      True. But Tor onions are servers.
      - duskwuff 7y ago
        Not in any sense that's relevant here. They don't host content.
        mirimir 7y ago
        I have no clue where you're coming from. What sort of content do you say Tor onions can't host? "Tor onion" just means that a server is (ideally) only reachable as an onion URL, which is only accessible via the Tor network. There is the limitation that Tor only handles TCP. Otherwise, one can route anything over Tor. In my experience, that includes HTTP(S), FTP, Tahoe-LAFS, SSH, RDP, Mumble, OpenVPN and tinc. And others, if I spent more time remembering what I've played with.
        duskwuff 7y ago
        Okay, I think I see the source of the confusion. What you're describing is a Tor hidden service. Hidden services are separate from the Tor relay network itself, which is what I thought you were referring to as "Tor onions". Hidden services are optimized for confidentiality over performance. Using them for bulk data storage would place a lot of load on the relay network, and it's not clear what security problem this arrangement would solve.
        mirimir 7y ago
        Again, sorry. As far as I know, "hidden service" is deprecated, with "onion service" the current term. And it does tend to get shortened to "onion". But I admit that it was confusing. Because relays used to be called "onion routers". Which is also more or less deprecated, I think. The security problem is Stores being physically located and compromised, based on IP addresses found in traffic logs.
      - sp332 7y ago
        I've never heard of an onion service called just "an onion" before. I don't know enough to say if that's wrong, just that I was confused.
        mirimir 7y ago
        Sorry to be confusing. It is commonly used, in some Tor communities. But yes, "onion service" is clearer.
mirimir 7y ago
What do you mean by "private"? I mean, if it's securely encrypted, does it matter if others can see it? And indeed, if it's online, you must assume that others can see it. My privacy concern is about IP addresses. So I'd want to use IPFS with Tor onions as stores.
- jerry292 7y ago
  Well I fear even if the information is encrypted alone once quantum computing breaks modern AES encryption standards that’s going to be a yikes. So I’d be more comfortable with encryption as well as access controls.
  - mirimir 7y ago
    I guess. But access controls really just keep the punters out. Any serious adversary will just track down the stores. And even if they're on dedicated servers with FDE, keys can be obtained from RAM.
    - brendoncarroll 7y ago
      Just to clarify: All data is encrypted on the client, going after a server backing a Store will get you encrypted blobs. Encryption keys would not exist on the server in plaintext.
      - mirimir 7y ago
        Sorry to confuse the issue. I meant the keys for the Store's FDE, not the WebFS keys, which never leave the user's machine. I was addressing jerry292's concern about access to the encrypted data in Stores.
  - brendoncarroll 7y ago
    This is a legitimate concern. WebFS is designed for the p2p storage use case. Persisting data with p2p storage means that it can live forever. All the secrets in WebFS are randomly generated and there are no user supplied (potentially weak) passwords. w.r.t. quantum computing: it is possible for WebFS to use symmetric cryptography for all remote data. Although, many Cell implementations in the near term will likely use elliptic curves or RSA.

toomim 7y ago

One issue that networked filesystems have is with mutations, especially with multiple writers. WebDAV, NFS, etc. try to address this with locking, but that doesn't allow simultaneous writes, which means it's dangerous to work offline. It's possible to solve this by using an OT or CRDT algorithm behind the scene. This is what we are building into an extension of HTTP called braid (https://braid.news https://braid.news). It could be useful for a web-backed filesystem. It automates synchronization.

layoutIfNeeded 7y ago
Thanks for the shameless plug, but what does this have to do with WebFS?

mirimir 7y ago

ELI5 please. Where is the data for all of those IPFS hashes stored? And what ensures that it will persist?

brendoncarroll 7y ago
WebFS stores data in things it calls "Stores". Stores can be anything that takes data and gives back a key to retrieve the data. Right now we have an IPFS and HTTP store built. An FTP server could also work as a store. If you or a friend run the HTTP or FTP server then it will persist the data for you. IPFS doesn't incentivize data persistence so if WebFS is working on top of IPFS it inherits that problem. You could run WebFS on top of one of the storage networks and persisting your blobs would be incentivized. WebFS is storage layer agnostic. Give it a Store, and it will give you a file system.
- mirimir 7y ago
  OK, cool. I understand now. So if WebFS is running on a system with access to Tor SocksPorts, can Stores be onion URLs? Edit: If not, one could presumably route WebFS through OnionCat's IPv6 /48. But that only works with v2 onions, which are deprecated. However, tinc works with v3 onions. And either of those gives you UDP transport.
  - brendoncarroll 7y ago
    Yes, you could use the HTTP Store over Tor. I think you would have to configure the proxy environment variables for Go's HTTP client.
    - mirimir 7y ago
      Thanks. I'll try it, and comment on GitHub.

rkagerer 7y ago

Your page mentions files in a store are identified by a hash of their contents. Does this have any privacy implications? Can another participant in the network tell what files are in my store? (like a party providing disk space, or someone intercepting my traffic)

brendoncarroll 7y ago
Encryption keys are derived from a secret and the hash of the data. The secrets are set per Volume. An empty secret would be a convergent encryption strategy, maybe suitable for public data. If the secret is set then the keys will be convergent with other keys generated with the same secret. This amounts to deduplication within a Volume, and privacy. All a storage provider sees are many small encrypted blobs, so the size of large files is not leaked either.

dredmorbius 7y ago

I'm going to assume you're familiar with Plan9OS's WebFS and 9P Protcol? https://en.m.wikipedia.org/wiki/9P_(protoco) https://en.m.wikipedia.org/wiki/9P_(protoco) I've been kicking around similar ideas (nowhere near implemented) for a while: https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if_the_web_was_filesystemaccessible/ https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if...

brendoncarroll 7y ago
I was not aware of Plan 9's WebFS. It looks like it presents websites in the local file system. IPFS can do something very similar with its content. I have used that before. The name here comes from using web resources (referred to by url) as building blocks for a file system. I guess "Web for FS" rather than "Web as FS".
- dredmorbius 7y ago
  The notion of remote services accessed via local filesystem dynamics is pretty well established. Among implementations: - NFS, particularly with the Solaris-originated concept of automounts over a /net mountpoint. - Various virtual filesystems. Midnight Commander ("mc") offers several of these, including archive formats (tar, cpio, afio, rpm, deb) and remote (FTP, SSH). - SMB/CFS/Samba - Various FUSE filesystems, including again ssh, ftp, and others. These generally require specifying in advance specific mountpoints. The notion of on-demand access to remote resources over protocols (e.g., http/https, or others), under filesystem dynamics, is interesting -- you can use any general tool, utility, or application for access, mediated through the filesystem by way of drivers, rather than a specific application (e.g., Web browser, FTP client, etc.) There are numerous issues. In particular, applications tend not to respond well to remote resources disappearing, changing, or failing to return from change requests -- NFS's behaviour with nonresponsive remote hosts is ... notorious. Consistency, availability, and partition resistance (CAP) are long-standing concerns, and there's no way to solve for all three. I'd add latency as another major consideration. https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem The general notion of managing and tracking changes locally, and pushing them to remote, has merits. I'm not aware of a "gitfs" ... though of course, one does exist, TIL: https://www.presslabs.com/code/gitfs/ https://www.presslabs.com/code/gitfs/ The notion of using git (or other versioning system) as a mediator for remote/local revisioned access seems to have merits. Obviously not viable for very-high-variability systems, but adequate for many occasionally-modified resources. I'm not sure if you're looking at using your WebFS itself as a publishing mechanism, though in general I think I'd recommend against doing this. For small-n peer-to-peer distribution that's probably workable, but for large-scale provisioning-and-request systems, relying on HTTP or other established transports is likely more sensible. One area I've recognised as being particularly fraught is the whole notion of security and privacy. Providing unfiltered local access to remote resources which may change arbitrarily is a great way for allowing malware onto local systems -- your transport layer should probably implement some level of security and mounts deny direct execution of content. The fact that remote content could be copied to an executable mountpoint remains, and would make numerous attacks possible. Similarly: access, update, write, and/or publishing actions all leak considerable information which could be of concern to specific users or organisations. Hash-based indexing (already addressed in this thread) being only one of several such vectors.

cbsmith 7y ago

Name is dangerously close to WebNFS.

mpweiher 7y ago

This looks really cool. > If you have ever thought "x can probably be used as a file system" ...you might also want to take a look at Storage Combinators[1]. Not quite the same problem space, but abstracting away a bit from both concrete filesystems and other storage mechanisms to get to a composable abstraction. Note: I am the primary author, and also taking a good look at WebFS for further inspiration... :-) [1] https://2019.splashcon.org/details/splash-2019-Onward-papers/7/Storage-Combinators https://2019.splashcon.org/details/splash-2019-Onward-papers...

ianopolous 7y ago

You will probably be interested in Peergos [0][1], which at it's lowest level is an encrypted global filesystem also built on IPFS and also only using the block api. [0] https://github.com/peergos/peergos https://github.com/peergos/peergos [1] https://book.peergos.org https://book.peergos.org

jerrygreenest 7y ago

I see very little docs and absolutely no fancy info like gifs or videos explaining what it's possible to do with this tool. I see some example folder, but I can't really say the difference between webfs and ipfs. (Just a little feedback, I mean no offence)