7 ms·
Willow Protocol
- j-pb 3y ago"Wrangling the complexity of distributed systems shouldn’t mean we trade away basic features like deletion, or accept data structures which can only grow without limit." The CALM theorem would like a word with you. You simply can't have consistent non-monotonic systems. Forgetting is ok, deleting is not.
- rudolph9 3y agoWhy would anyone use this over libp2p?
- chadk 3y agoIf you are looking for more on Willow, we had gwil over for a chat at one of our tech talks: https://www.youtube.com/watch?v=yx5T7Z5rHGc https://www.youtube.com/watch?v=yx5T7Z5rHGc
- catapart 3y agoSo this is pure spec? No implementations at all?
- castles 3y agohttps://willowprotocol.org/more/projects-and-communities/index.html#projects_and_communities https://willowprotocol.org/more/projects-and-communities/ind...
- nathan_phoenix 3y agoStill confused to be honest... What does it mean that Earthstar will become a Willow protocol? Isn't it an implementation of Willow?
- m3talsmith 3y agoThere are at least two implementations there on that page: - One in typescript - One in rust
- candiddevmike 3y agoA spec without an implementation is a lovely idea at best.
- rklaehn 3y agoiroh developer here. willow was not developed in a vacuum. the willow folks have worked with us while we have implemented many ideas from willow, starting with range based set reconciliation ( https://arxiv.org/abs/2212.13567 https://arxiv.org/abs/2212.13567 ) they have been open to removing parts that have turned out to add too much complexity to implementations.
- b_fiive 3y agoiroh documents are a work-in-progress implementation of willow: https://github.com/n0-computer/iroh https://github.com/n0-computer/iroh We've been working with the willow team as we go & giving feedback on the spec. disclosure: I work on iroh.
- elmolino89 3y agoThe iroh name resembles a bit iRODS, another system for distributed file sharing and fine grained permissions. Quick googling did not give me a proper grasp of the use cases for iroh/IPFS vs iRODs. Would you be willing to list the benefits of iroh vs IRODS?
- b_fiive 3y agowow, thank you for pointing me to IRODS, I was not aware of the project! Big difference I'm seeing as I read the docs for IRODs is a datacenter-grade data management _service_, whereas iroh is a multiplatform SDK for building your own applications. Seems like one would want IRODs if they have massive amounts of highly sensitive data that needs fine grained access control. You would want iroh if you're building an app that uses direct connections between end-user devices to scale data sync
- rklaehn 3y agoI was not aware of iRODS. Iroh is named by a certain fictional character that likes tea. Any similarity is a coincidence. But it seems like iRODS is much more high level than iroh. E.g. iroh certainly does not contain anything for workflow automation. You could probably implement something like iRODS using iroh-net and iroh-bytes.
- jw_cook 3y ago> Iroh is named by a certain fictional character that likes tea. "The file was in my sleeve the whole time!"
- tux3 3y agoHow does this compare to IPFS? I personally found IPFS very disappointing in practice, so I'm very hopeful for a successor. (The promise of IPFS is great, but it is excruciatingly slow, clunky, and buggy. IPFS has a lot of big ideas but suffers from a lack of polish that would make Augías look clean. And as soon as you scale to larger collections of files, it quickly crumbles under its own weight. You can throw more resources at it, but past some point it just falls over. It just doesn't work outside of small-scale tests.)
- anacrolix 3y agoConsider https://github.com/anacrolix/btlink https://github.com/anacrolix/btlink. It's a proof of concept, and has all the basics. I designed it and I worked for IPFS, and I am the maintainer of a popular DHT and BitTorrent client implementation.
- pera 3y agohttps://willowprotocol.org/more/compare/index.html#willow_compared https://willowprotocol.org/more/compare/index.html#willow_co...
- rklaehn 3y agoif you are looking for something similar to ipfs but a bit more minimalistic and performance oriented, check out iroh https://github.com/n0-computer/iroh https://github.com/n0-computer/iroh . It is a set of open source libraries for peer to peer networking and content-addressed storage. It is written in rust, but we have bindings to many languages. One part of iroh is a work in progress implementation of the willow spec. The lower layers include a networking library similar to libp2p and a library for content-addressed storage and replication based on blake3 verified streaming. Most iroh developers have been active in the ipfs community for many years and have shared similar frustrations... See this talk from me in 2019 :-) https://youtu.be/Qzu0xtCT-R0?t=169 https://youtu.be/Qzu0xtCT-R0?t=169
- nobodywillobsrv 3y agoIs iroh suitable for use as a collaborative decentralized function cache? Had a quick skim but can't quite see how to get at the get/put etc in the python api yet. Will spend more time later.
- wavemode 3y ago> Some questions in protocol design have no clear-cut answer. Should namespaces be identified via human-readable strings, or via the public keys of some digital signature scheme? That depends entirely on the use-case. To sidestep such questions, the Willow data model is generic over certain choices of parameters. You can instantiate Willow to use strings as the identifiers of namespaces, or you could have it use 256 bit integers, or urls, or iris scans, etc. > This makes Willow a higher-order protocol: you supply a set of specific choices for its parameters, and in return you get a concrete protocol that you can then use. If different systems instantiate Willow with non-equal parameters, the results will not be interoperable, even though both systems use Willow. Help me out here - isn't the point of a protocol that two independently developed systems don't have to agree on how to implement the protocol? What value does Willow have if two systems that both purport to be "Willow-compatible" aren't compatible with each another?
- cobertos 3y agoBy having a more generic protocol on top, it allows the same tools to be used for different specific end results. So shared libraries and common debugging tools can benefit more use cases. You might even make a "higher-order" tool that can work with any willow data, at the cost of specific UI affordances that can be used when you know more about the underlying data. This is sort of how ActivityPub is a thing, but it underpins multiple, sorta-but-not-really interoperable systems like Lemmy and Mastodon.
- smaudet 3y agoNo I think you are right, this isn't a protocol. Its a protocol generator... "higher order" is some nonsense, and would make me shy away from using it...
- gray_-_wolf 3y agoIt is unusual term in the area of protocols, but it seems understandable it tries to draw a parallel to higher order functions. So "some nonsense" might be a bit strong...
- helpfulContrib 3y ago[dead]
- throwaway2562 3y agoStill confused here. What is the actual, concrete ‘as-a-user-I-want-to’ application for which this is meant to be an ideal fit? Sorry if a dumb question.
- bo1024 3y agoYeah, it would have helped me if they walked through what it actually means to "use" Willow. Do I install something like Dropbox on my computer? Do I write code that calls Willow as a library?
- layer8 3y agoIt’s like Dropbox, including sharing, but without a centralized service, instead peer-to-peer.
- dirkf 3y agoSo something like https://syncthing.net/ https://syncthing.net/ ?
- layer8 3y agoSyncthing doesn’t have sharing support AFAICS, but yes, Willow could be used as the underlying protocol for something like Syncthing.
- rklaehn 3y agoIt's more generic than that. Syncthing is designed specifically for file system sync (and does a very good job). Willow could be used for file system tasks, but also for storing app data that is unrelated to file systems, like a KV store database. You should be able to write a good syncthing like app using the willow protocol, especially if you choose blake3 as the hash function.
- jauntywundrkind 3y agoIt should be able to underpin most apps. The sky is the limit. Or your imagination is the limit; whichever comes first. This is a protocol for generic shared information spaces, where each person still owns & can manage permissions for their pieces of data in the space. It's a general idea that's present & implicit in most existing online spaces.
- jhardy54 3y agoAljoscha and gwil are excellent people, I’m excited to see them working together. Looks to me like they’re solving some of the biggest problems with Secure Scuttlebutt.
- cranberryturkey 3y agohttps://reddit.com/r/willowprotocol https://reddit.com/r/willowprotocol
- philsnow 3y ago> Unreviewed Content > This community has not been reviewed and might contain content inappropriate for certain viewers. View in the Reddit app to continue. Wow, what absolute horseshit. The march continues to acquire marketing signals at any cost.
- ynniv 3y agoTotal erasure of data. This is disappointing. What's been read can never be un-read; to say otherwise is deceptive.
- black_puppydog 3y agoIn the same way, once an attacker can exploit some weakness in a system, it's game over. Yet defense in depth is a thing and makes it much less likely that bad things happen. In this case, yes, it's impossible to guarantee that some malicious peer doesn't ignore my "plea to delete". But combined with the fact that my data will only be replicated to/by peers I already have a trust relationship with (as opposed to e.g. on a blockchain) it provides another layer of protection that a system without deletion simply doesn't have. Not perfect, but not useless either.
- ynniv 3y agoYes, but that's not what "Total erasure of data" means. The project's goals are hard and noble. It would be better to under-promise and over-deliver than to make everyone question their claims. Maybe I'm just a grumpy old man at this point, but there are already too many caveat emptors in computing. They could have said "better" erasure of data.
- wharvle 3y agoIt's a handy feature to have in the protocol if you're operating detached networks using it, and can control all the clients. If you're using it as internal infrastructure. Which, personally, is the only way I've ever been interested in using these sorts of things. You're right that it's nigh-meaningless for a public cluster.
- ynniv 3y agoOk, but where exactly is that when the FBI is pulling Bitcoin private keys out of files accidentally synchronized to iCloud Files? These are hard problems.
- randall 3y agoSo stoked about this. Lower level than holepunch, and it sounds like it has everything I need to get going.
- pmarreck 3y agomy first question (given that IPFS doesn’t seem to do this well) is “does it scale?” Still useful otherwise if not… assuming there’s an actual client/server for it on mac/linux…
- rklaehn 3y agoThis is a good question. But it is worth noting that not everything has to scale globally. E.g. in iroh-sync (which is an experimental impl of the willow protocol) you are not concerned with global scaling. You care only about nodes that are in the same document. So while if you request hash QmciUVE1BqKPXMSvTTGwHZo1ywYdZRm9FfBvEJkB6J4USb via ipfs, you are trying to globally find anybody that has this hash, which is a very difficult task. If you ask for some content-addressed data in an iroh document, you know to only ask nodes that participate in this particular document, which makes the task much easier. Edit: regarding clients, iroh is released for osx, windows and linux. Iroh as a library also works on ios. Download instructions are here: https://iroh.computer/docs/install https://iroh.computer/docs/install
- waterheater 3y agoAnother webpage compares Willow to other protocols like IPFS: https://willowprotocol.org/more/compare/index.html#willow_compared https://willowprotocol.org/more/compare/index.html#willow_co... According to them, data on IPFS is immutable, stateless, and globally-namespaced, whereas data on Willow is mutable, stateful, and conditionally-namespaced. I interpret Willow as an authenticated, permissioned, content-based, globally-addressed, distributed database system, where an address has the hierarchy and expressiveness of a URL. One particularly nice feature about the documentation: if you hover over an underlined word (https://willowprotocol.org/specs/data-model/index.html#data_model https://willowprotocol.org/specs/data-model/index.html#data_...), a pop-up box provides a definition or explanation. Importantly, some terms in the pop-up are underlined themselves, so you can dig down into the terminology with great ease. More projects should implement this functionality.
- Tillerino 3y agoI love the style of the documentation. That hover feature is just great. _Surely_, they didn't write this from scratch, did they? _Surely_, there is a tool that they used that I can use, too. Right?
- Tillerino 3y agoI asked. They made their own generator which they want to release once it is polished up a bit: https://github.com/earthstar-project/willowprotocol.org https://github.com/earthstar-project/willowprotocol.org
- Kinrany 3y agoWhat's the purpose of subspaces, given that there are namespaces? What's the purpose of having separators in the keys?
- still_grokking 3y ago[flagged]
- orthecreedence 3y agoThis is kind of exactly what I've been looking for. I've been trying to weave stuff together with libp2p, but this looks very promising as a way to handle a lot of the lower-level junk I don't care about. While I didn't go in-depth on the docs, I can see that this would be able to model a lot of different applications right off the bat. Very cool.
- jd3 3y agoother commenters have mentioned ipfs, dropbox, syncthing, etc. but this most closely resembles http://upspin.io/ http://upspin.io/ with the caveat that willow is p2p and upspin uses a centralized key server https://www.youtube.com/watch?v=ENLWEfi0Tkg https://www.youtube.com/watch?v=ENLWEfi0Tkg
- KomoD 3y agoI really like the illustrations
- m3kw9 3y agoDecentralized and no ICO needed
- snvzz 3y ago>Willow specification published 17/01/2024 Please try to follow RFC3339 when writing dates. E.g. 11/01/2024 is ambiguous, as it could be January 11th or November 1st, whereas 2024-01-11 is RFC3339-compliant and does not exhibit this problem.