4 ms·

I spent a lot of time with photos (Picasa) trying to do this peer to peer - this is what we built in the 2002 era. Here are a few issues: 1. Identity is hard t

by herf 7y ago

I spent a lot of time with photos (Picasa) trying to do this peer to peer - this is what we built in the 2002 era. Here are a few issues: 1. Identity is hard to do on the LAN, so any sharing ends up using the cloud to figure out who has access. Similarly, identity is hard to move around, so representing your Facebook comments feed outside Facebook is difficult to do. 2. Any time you have a "special" server that handles identity, merging, or any other task, it ends up in the effective role of master, even if the rest of the parts of your system are peers. You want all your collaborations in Dropbox to survive their infrastructure being offline? It's tough to do. 3. p2p stalled a bit in the mid-2000s when storage and bandwidth got much cheaper--in a period of just two years (2002-2004), it became 100x cheaper to run a cloud service. But what continued to stall p2p was mobile. Uploading and syncing needs to run in the background, and if you're on a limited bandwidth client or a battery-limited device like iOS, sync can effectively diverge for months because the applications can't run in the background. So changes you "thought" you made don't show up on other devices. 4. For avoiding mass surveillance, what we are missing from this older time is the ability to make point to point connections (between peers) and encrypt them with forward secrecy, without data at rest in the cloud. Even systems that try to do some encryption for data at rest (e.g., iMessage) keep keys essentially forever, so data can be decrypted if you recover a private key later on. A system that only makes direct connections between peers does not have this issue. 5. Anytime you have multiple peers, you have to worry about old versions (bugs), or even attackers in the network, so it's fundamentally harder than having a shared server that's always up to date and running the latest code.

koheripbal 7y ago
With regards to identity management, maybe there should be a formalized integration between browsers and password managers such that the concept of "registration" goes away and new logins just automatically create user accounts with default permissions, according to email address.
- mceachen 7y ago
  Centralized identity management was the grand promise of OAuth. It seems that somehow ended up, in most practical applications, as a "login with Facebook" button. There are a number of startups working on simplified or "passwordless" auth, but it seems that none have substantive traction. I'd love to be proven wrong, here, though!
  - voxic11 7y ago
    The Web Authentication API is already implemented in the major browsers (excluding old IE and Safari). https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Authent...
  - bdcravens 7y ago
    Before that there was Microsoft Passport. Federated login for the web is a “problem” various parties have been working on for 20 years.
    - jandrese 7y ago
      There's a fundamental problem of "I want my credentials to be validated by some central service, but I don't want to give some big faceless company my information." The centralized login service can be used to track your activity all across the net. It's a lot of power to give to some people you don't know. Worse, people's fears in this area are completely justified. Precious few companies have proven themselves to be contentious with our personal data. Some go as far as to repackage and outright sell said data for personal gain. A decentralized blockchain-like system might work for this, but as far as I know it has not been attempted.
      - dane-pgp 7y ago
        Actually, the problem of centralized login services tracking users across the net was considered and partially solved in 2010 by a team from Google and MIT. This was at a time when OpenID was the most popular federated login system, so the proposal was called PseudoID. Unfortunately this idea didn't gain more traction, and there are only a few references to it online now, such as this research paper and a YouTube video by one of the authors: https://ai.google/research/pubs/pub36553 https://ai.google/research/pubs/pub36553 https://www.youtube.com/watch?v=fCBPuGsO_I4 https://www.youtube.com/watch?v=fCBPuGsO_I4 Also, it didn't address all the methods by which a malicious identity provider could track the user, so it would probably have to be extended by having support added in the browser.
- thinkmassive 7y ago
  For true P2P identity there has been research happening for years on distributed PKI, resulting in specifications like DID[1] and the Sidetree[2] protocol. 1: https://w3c.github.io/did-core/ https://w3c.github.io/did-core/ 2: https://github.com/decentralized-identity/sidetree/blob/master/docs/protocol.md https://github.com/decentralized-identity/sidetree/blob/mast...
nihonde 7y ago
This is fantastic first-hand feedback. Thanks for sharing this.
mceachen 7y ago
Thanks for taking the time to share this! The issue I face with PhotoStructure is that people's home network frequently has throttled upstream rates. I'd love to provide a caching CDN, but I want all content encrypted. I don't want my pipes to see any data from my users. How would you do perfect forward secrecy when only the library owner's key is available at upload time? Is it possible? If not, it seems that every bit of shared content would have to be re-encrypted and re-uploaded when someone new is granted access to that content.
- sounds 7y ago
  One option is an envelope keys. Content is encrypted and published by the owner, but the owner uses a new random key to do the encryption. This new random key is the "envelope key." The owner then takes the envelope key, encrypts it with the public key of recipient A, and publishes the encrypted message containing the envelope key. (This message with the envelope key should also be signed by the owner using the owner's private key.) Anyone who obtains the envelope key effectively has read-only access to the content. The owner can publish a separate message with the envelope key, all these messages have recipient B, C, and D's different public keys, so if you are a recipient you will want to know which message is likely to be encrypted by your public key, after which you can decrypt it to get the envelope key. The owner also needs to sign the content. Signing the content and later verifying the content is done with the owner's public/private keypair, not the envelope key. A glaring weakness of envelope keys is that the recipient can share the content freely once they have it. There's no content protection after the content and the envelope key are published. The recipient can easily just share the envelope key if they want to. But they can also just copy the content (once they have decrypted it).
  - Spearchucker 7y ago
    This gets complex fast. The access control list needs to travel with the data. Revocation can theoretically take forever (until all offline clients get back online). Onwer signing content is great but breaks rule 4 (It should support collaboration). I've been working on my own personal-use app for my twin and I to collaborate on documents from different continents so pretty close to the problem.
masukomi 7y ago
SSB (Secure Scuttlebutt)[1] has solved all of these problems except the encryption of data using the same key forever. The apps all currently use one identity per device but it's not actually hard to use the same identity on two devices and multiple apps share the same identity commonly. Old versions could of software could be a concern but really that's what Versioned APIs are for. It's a solved problem. The only notable problem with local first dev that I'm aware of currently is storage requirements being confounded by the limited size of SSDs on most people's computers. [1]: https://scuttlebutt.nz/ https://scuttlebutt.nz/
3xblah 7y ago
"I spent a lot of time with photos (Picasa) trying to do peer to peer..." As I recall, one of Google's many "failed" projects was a means for transferring large numbers of photos person to person called "HELLO". I reckon this existed for a short time post-2004, then AFAIK disappeared from public view (or morphed into something else). I could be remembering the details incorrectly.

by herf 7y ago

koheripbal 7y ago

With regards to identity management, maybe there should be a formalized integration between browsers and password managers such that the concept of "registration" goes away and new logins just automatically create user accounts with default permissions, according to email address.

mceachen 7y ago
Centralized identity management was the grand promise of OAuth. It seems that somehow ended up, in most practical applications, as a "login with Facebook" button. There are a number of startups working on simplified or "passwordless" auth, but it seems that none have substantive traction. I'd love to be proven wrong, here, though!
- voxic11 7y ago
  The Web Authentication API is already implemented in the major browsers (excluding old IE and Safari). https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Authent...
- bdcravens 7y ago
  Before that there was Microsoft Passport. Federated login for the web is a “problem” various parties have been working on for 20 years.
  - jandrese 7y ago
    There's a fundamental problem of "I want my credentials to be validated by some central service, but I don't want to give some big faceless company my information." The centralized login service can be used to track your activity all across the net. It's a lot of power to give to some people you don't know. Worse, people's fears in this area are completely justified. Precious few companies have proven themselves to be contentious with our personal data. Some go as far as to repackage and outright sell said data for personal gain. A decentralized blockchain-like system might work for this, but as far as I know it has not been attempted.
    - dane-pgp 7y ago
      Actually, the problem of centralized login services tracking users across the net was considered and partially solved in 2010 by a team from Google and MIT. This was at a time when OpenID was the most popular federated login system, so the proposal was called PseudoID. Unfortunately this idea didn't gain more traction, and there are only a few references to it online now, such as this research paper and a YouTube video by one of the authors: https://ai.google/research/pubs/pub36553 https://ai.google/research/pubs/pub36553 https://www.youtube.com/watch?v=fCBPuGsO_I4 https://www.youtube.com/watch?v=fCBPuGsO_I4 Also, it didn't address all the methods by which a malicious identity provider could track the user, so it would probably have to be extended by having support added in the browser.
thinkmassive 7y ago
For true P2P identity there has been research happening for years on distributed PKI, resulting in specifications like DID[1] and the Sidetree[2] protocol. 1: https://w3c.github.io/did-core/ https://w3c.github.io/did-core/ 2: https://github.com/decentralized-identity/sidetree/blob/master/docs/protocol.md https://github.com/decentralized-identity/sidetree/blob/mast...

nihonde 7y ago

This is fantastic first-hand feedback. Thanks for sharing this.

mceachen 7y ago

Thanks for taking the time to share this! The issue I face with PhotoStructure is that people's home network frequently has throttled upstream rates. I'd love to provide a caching CDN, but I want all content encrypted. I don't want my pipes to see any data from my users. How would you do perfect forward secrecy when only the library owner's key is available at upload time? Is it possible? If not, it seems that every bit of shared content would have to be re-encrypted and re-uploaded when someone new is granted access to that content.

sounds 7y ago
One option is an envelope keys. Content is encrypted and published by the owner, but the owner uses a new random key to do the encryption. This new random key is the "envelope key." The owner then takes the envelope key, encrypts it with the public key of recipient A, and publishes the encrypted message containing the envelope key. (This message with the envelope key should also be signed by the owner using the owner's private key.) Anyone who obtains the envelope key effectively has read-only access to the content. The owner can publish a separate message with the envelope key, all these messages have recipient B, C, and D's different public keys, so if you are a recipient you will want to know which message is likely to be encrypted by your public key, after which you can decrypt it to get the envelope key. The owner also needs to sign the content. Signing the content and later verifying the content is done with the owner's public/private keypair, not the envelope key. A glaring weakness of envelope keys is that the recipient can share the content freely once they have it. There's no content protection after the content and the envelope key are published. The recipient can easily just share the envelope key if they want to. But they can also just copy the content (once they have decrypted it).
- Spearchucker 7y ago
  This gets complex fast. The access control list needs to travel with the data. Revocation can theoretically take forever (until all offline clients get back online). Onwer signing content is great but breaks rule 4 (It should support collaboration). I've been working on my own personal-use app for my twin and I to collaborate on documents from different continents so pretty close to the problem.

masukomi 7y ago

SSB (Secure Scuttlebutt)[1] has solved all of these problems except the encryption of data using the same key forever. The apps all currently use one identity per device but it's not actually hard to use the same identity on two devices and multiple apps share the same identity commonly. Old versions could of software could be a concern but really that's what Versioned APIs are for. It's a solved problem. The only notable problem with local first dev that I'm aware of currently is storage requirements being confounded by the limited size of SSDs on most people's computers. [1]: https://scuttlebutt.nz/ https://scuttlebutt.nz/

3xblah 7y ago

"I spent a lot of time with photos (Picasa) trying to do peer to peer..." As I recall, one of Google's many "failed" projects was a means for transferring large numbers of photos person to person called "HELLO". I reckon this existed for a short time post-2004, then AFAIK disappeared from public view (or morphed into something else). I could be remembering the details incorrectly.