6 ms·

Braid: Synchronization for HTTP

233 points by walterbell 2y ago

victorbjorklund 2y ago
I dont get it. What would I use this for? (Honest question. Im probably not smart enough to get it)
- chronicler 2y ago
  same reason you'd use HTTP, it's a superset of HTTP
  - toomim 2y ago
    Yes, and to elaborate — for any HTTP usage with dynamic state. HTTP was invented for static pages, that were written by hand, and rarely changed. But today's web has dynamic pages, driven by javascript and databases, and users expect them to update in realtime. Braid-HTTP adds support for dynamic state to HTTP.
  - victorbjorklund 2y ago
    I still dont get it. Is this to sync two servers between each other?
    - whilenot-dev 2y ago
      From the article: > Together, these features enable a web resource to synchronize automatically across multiple clients, servers and proxies, and support arbitrary simultaneous edits by multiple writers, under arbitrary network delays and partitions, while guaranteeing consistency using an OT, CRDT, or other algorithm. So, you'd use it if you want to "support arbitrary simultaneous edits by multiple writers, under arbitrary network delays and partitions, while guaranteeing consistency", for example for apps that want to support collaborative editing. Braid wants to turn state transfer (like in RESTful API designs - Representational State Transfer) into a kind of synchronization transfer for state. Currently, state synchronization is handled on the state management layer of each individual application, and API calls just transfer that state, but Braid wants to have it solved as an extension to HTTP libraries on the API layer of an application AFAICS.
      - toomim 2y ago
        Yes, or even more simply: Let's say that your client runs GET /some-json, and that JSON gets updated, and the client wants to get the updates. Right now, your options are: 1. Re-run the GET /some-json again from the client (polling) 2. Start a websocket, and invent some custom protocol over the websocket to let the client subscribe to /some-json. With Braid, you just do: GET /some-json Subscribe: true And the client will automatically get the updates, within HTTP. The Braid-HTTP ponyfill library abstracts the details behind a normal fetch() for you, until browsers implement it. Today, we tend to use HTTP for static assets, and then switch to a websocket or SSE or something whenever things get dynamic. That sucks! We lose the standard of HTTP! Braid lets you keep using HTTP, even for your dynamic assets. It also solves issues in caching. Instead of using max-age heuristics, we have actual real versioning! And yes— the protocol generalizes all the way to full-peer-to-peer multi-writer CRDT/OT algorithms. But start simple!
        whilenot-dev 2y ago
        Frankly, your explanation isn't any simpler, quite the contrary. > That sucks! We lose the standard of HTTP! Why does this suck? WebSockets or SSEs are also standardized. > the protocol generalizes all the way to full-peer-to-peer multi-writer Why "peer-to-peer"? While Braid can be used in a local-first setting and probably do peer-to-peer, that isn't the focus here. The focus is to help application developers to not get into the shenanigans of CRDTs and distributed algorithms etc., but solve those things already via polyfills.
        toomim 2y ago
        WebSockets and SSE are standards in the same way that TCP is a standard -- if you use them, you are still defining an ad-hoc protocol on top to subscribe to state and publish new state. HTTP is a standard on top of TCP. It provides a higher-level abstraction than a socket -- the abstraction of State Transfer. When you use a WebSocket, you're back to a low-level socket, and have to redefine "state", and the methods to get it, put it, and subscribe to it. Since each web programmer defines those methods in a different way, his state gets hidden behind his own non-standard protocol. There is no way for website A to re-use the state on website B, for instance, without learning website B's custom WebSocket protocol and re-implementing it perfectly. CDNs and other caches cannot handle WebSocket traffic. But if you use a standard like Braid-HTTP, they can cache your dynamic assets along with your static assets.
    - toomim 2y ago
      You could sync two servers. Or you could sync a server with clients. Braid is transitioning HTTP into a peer-to-peer world, where the distinction between clients and servers doesn't matter.
      - victorbjorklund 2y ago
        Could you call it websockets without websockets?
        toomim 2y ago
        In some sense, you could! The reason is that almost every use WebSockets is actually for Synchronizing State. What you really want to do is to synchronize state. Braid is a protocol for doing just that. So you don't need to turn to WebSockets anymore!
        victorbjorklund 2y ago
        I wonder if Braid could be a transport layer for Phoenix Liveview. It uses Websockets but falls back to long-polling if websockets isnt available.
        toomim 2y ago
        Yes! Phoenix Liveview is very similar to the Statebus project (https://stateb.us https://stateb.us) that inspired Braid. Braid was the protocol that Statebus needed, and Phoneix Liveview is one of the most exciting projects in the Statebus space that I know! I think we are working from a common inspiration!
- laminatedsmore 2y ago
  I didn't get this either until I looked at the linked doc, which breaks it down more obviously imo (section 6.1 for examples). Looks interesting to me, seems nice to be able to handle refreshing content with pure http as described here rather than with web sockets. https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http#section-6.1 https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
- sesm 2y ago
  From what I understood, it's a protocol for exposing a tree of files for synchronization and concurrent editing, similar to WebDAV.
bawolff 2y ago
I cant help but think that this feels like webDav on steriods, and webdav never really caught on. I kind of think sometimes it makes more sense to layer on top of http instead of extending.
- mike_hearn 2y ago
  It kind of is layered on top, no? WebDAV didn't really catch on, but the general product space of remote drives did. The problem with WebDAV is that like most Web* tech (and maybe Braid) it's design by committee in the abstract. To make remote drives work well for the end user requires a fairly complicated protocol with tons of ugly edge cases. The interoperable standards from committees approach tends to fail in those situations, whereas hard driving startups that use proprietary protocols they can iterate quickly tend to win. Iteration speed > openness, when there's nothing to copy from. Standardization tends to be about bigger companies trying to commodify their competitors.
  - 0x006A 2y ago
    many proprietary protocols iterate themself out of existence, just because some of them survive does not mean its the better model. Look at the video codec space, proprietary codes have long lost and have gone the way of Real Media, Windows Media Video and many others. While H.26X which is the result of design by committee are used everywhere.
    - treyd 2y ago
      It might be a different scenario with (modern) codecs where to get good use out of them it helps a lot to ship dedicated hardware. That's not true of file sync protocols.
  - giovannibonetti 2y ago
    > Standardization tends to be about bigger companies trying to commodify their competitors. Didn't you mean to commodity their complements? Like when Microsoft (software company) commoditied the hardware on top of where their OS ran.
dotancohen 2y ago
Interesting idea, I would certainly have use for it. But the current severside library support is limited to Lua and Haskell - quite the sign that the project is squarely in the hands of academics and has never seen real world dependent on a heavily used system. Even if my stack were based on a supported language, I would take pause at that.
- lionkor 2y ago
  luajit is stupid fast, and cooperating with a lua program from other languages is trivial. Might be worth a look?
  - dotancohen 2y ago
    It is certainly worth a look, and of course if the project takes off then Python and other library support would be expected eventually. That's how open source communities usually develop. My point was that the current library support is a good proxy for the state of development and deployment, and currently this state is very early.
    - mst 2y ago
      Given there's both client and server code available for JS that seems like a perfectly reasonable place to start experimenting with it. The python community tends to be less neophilic than JS (often to its advantage) so I wouldn't expect that to show up until later.
- josephg 2y ago
  There are a couple nodejs implementations as well. I would know - I wrote one of them.
  - junto 2y ago
    This looks somewhat similar to websockets and SignalR. What am I missing here? Is it like GRPC as a protocol, built on top of HTTP/2? I’d be interested in working on a port for .NET.
BigParm 2y ago
Can someone ELIF? I can synchronize clients around some state using http. What is an http extension? Does it add new http methods? If so, will anyone else be able to respond to these methods? Isn't http stateless what do you mean versioning? And etcetera! I'm sorry to criticize but who is this article written for? It's either so dense that you already need to be an expert to unpack all the information, or the information is not there. I would recommend starting from the assumption that the Hacker News layman knows the http protocol. Make sure what you show us is possible to follow from some reasonable starting point.
- toomim 2y ago
  HTTP (HyperText Transfer Protocol) is a State Transfer protocol. ReST (Representational State Transfer) is a State Transfer architecture. It was built for transferring a page from server to client. But if the page changed afterward, the protocol threw up its hands, and left it to the user to click "reload." Braid proposes new HTTP headers that give HTTP new features, so that instead of just being a State Transfer protocol, it gains the functionality to become a full State Synchronization protocol. To "extend" a protocol means to add new features to it, e.g. with new headers, which doesn't break existing uses of the protocol, but allows new implementations to opt into new extended functionality. HTTP was originally stateless, but then added cookies. Over time, web apps evolved to store tons of state in databases on the server. Then they started storing tons of state in javascript variables and DOM on the client. Today, most of the code we write in a web app is there to synchronize the changing state on a server with the changing DOM on the client. The web is no longer static pages. It is dynamic state. But since we haven't extended HTTP to support dynamic state, programmers have to write all this synchronization code by hand. This has been a pain in the ass, and so we've evolved monstrous stacks of Javascript frameworks (react, redux, etc. etc.) to try to help manage this state synchronization from server to client. If we build this functionality directly into HTTP, tons of code goes away, and the web itself gains great new features. Then, since we will have a standard for state synchronization, our state can interoperate, and walled siloes of websites break down. We can have a P2P web. When this work is complete, the browser will no longer need a reload button. The browser/server will guarantee that every page is always up-to-date, and every peer that synchronizes with that state will have an equal copy.
toomim 2y ago
We're about to release a new braid-text library: https://github.com/braid-org/braid-text/ https://www.npmjs.com/package/braid-text This is the easiest way to add collaborative editing to a web app. You can add it to any (req, res) handler in your nodejs app. No websocket necessary, since it extends HTTP itself! Plus, it features a new `simpleton` merge-type that requires zero history overhead on the client, and you can implement the client protocol from scratch in just 50 lines of code. There's no reason not to add collaborative editing to every user-editable string in your app now! All the network traffic is just HTTP, with an open, easy-to-read protocol that supports any OT or CRDT (currently defaulting to josephg's Diamond-Types). You can even read and write directly using the Braid-Chrome extension, which adds a devtools panel to view the version history of any Braid-HTTP resource: https://github.com/braid-org/braid-chrome https://github.com/braid-org/braid-chrome. I'm really excited about this library. We were going to announce it tomorrow (at https://braid.org/meeting-86 https://braid.org/meeting-86), but since this hacker news discussion is happening... I can't hold myself back from talking about it now!
- notpushkin 2y ago
  Neat! Any plans to release a Firefox extension, too? It shouldn't be too hard, IIRC the API is largely compatible.
  - toomim 2y ago
    That is cool! Nobody has worked on a FireFox devtool yet, but we welcome contributions.
- remram 2y ago
  > No websocket necessary, since it extends HTTP itself! What does that mean? How do other clients get notified of remote changes? edit: Braid include its own SSE-like construct, completely incompatible, with a brand-new HTTP status code "209". Seems unnecessary... How does that work with existing servers/proxies/middleboxes?
  - toomim 2y ago
halayli 2y ago
HTTP is a request-response protocol. It doesn't deal with state transfer nor does it define it. I find it weird that the first thing they mention is this: > Braid-HTTP is an extension to HTTP that generalizes it from a state transfer to a state synchronization protocol. They could simply say it's a synchronization protocol over HTTP.
- toomim 2y ago
  > HTTP ... doesn't deal with state transfer nor does it define it. The HTTP acronym stands for HyperText Transfer Protocol. ReST stands for Representational State Transfer. HTTP began as a HTML Transfer Protocol, but then generalized to content-types beyond HTML, such as images, scripts -- and general state. So today HTTP is known as a State Transfer Protocol. This architecture was canonicalized as ReST in Roy Fielding's dissertation. In order to generalize HTTP from State Transfer to State Synchronization, we have to augment some parts of it, such as request/response. Instead of just getting a single response to a request, Braid allows a client to subscribe to a resource, with a single request which will be then given multiple responses -- one response for each update to the resource. It is quite elegant to extend HTTP at this level! And we can do so without requiring any changes to web browsers.
EternalFury 2y ago
I wonder how well that scales and how it handles network partitions or erratic peers.
- toomim 2y ago
  You can use CRDT algorithms with Braid that heal from network partitions transparently, and guarantee strong eventual consistency. Here's an example CRDT algorithm for collaborative text that can handle arbitrary network partitions, and guarantee full eventual consistency, and prune all unnecessary history: https://braid.org/antimatter https://braid.org/antimatter This algorithm is the first to do so, and happened to be developed in the Braid group. As for scaling, check out JosephG's https://josephg.com/blog/crdts-go-brrr/ https://josephg.com/blog/crdts-go-brrr/ CRDT, which scales great, and is thoroughly tested. The Braid protocol is also architected with a new type of OT/CRDT architecture called a Time Machine that lets you do advanced things like apply backpressure through a network to decrease the frequency of updates, which we presented in https://braid.org/meeting-81 https://braid.org/meeting-81, and are releasing in the https://github.com/braid-org/braid-text https://github.com/braid-org/braid-text library that you can try right now.
superkuh 2y ago
For an HTTP company they sure do hate hypertext. Their entire write-up is just a blank white page unless one successfully executes the javascript from 5 separate domains.
- toomim 2y ago
  Point taken. But to be fair, we're not a company, but a working group. And we don't hate hypertext, we just haven't gotten around to implementing server-side rendering, which would be a nice thing to do.
  - superkuh 2y ago
    Glad to hear it. What on that page of just text and images made you feel an entire application was appropriate instead of just a hypertext document? I hope this application-centric approach isn't also being applied to the HTTP extension. Hypertext documents should at least get an equal share of the consideration for an HTTP protocol. I get that for commerce HTTP is just a transport to deliver the javascript/json/etc application. That's the way things are. But HTTP in general has to handle more use cases than just commerce. I worry that adding this dynamic checking of state to HTTP itself will make websites that adopt it inaccessible and unusable for almost every browser that currently exists. Only new software will be able to use them. It's a substantial break. Maybe don't call it HTTP.
skybrian 2y ago
They seem to have made a choice that URL's do not include version numbers. Instead, the version is sent as a separate header. That would make it hard to link to a specific version. I suppose there's nothing stopping anyone from also having URL's for specific versions, but it's not standardized. More generally, I'm unconvinced that synchronization should be so closely tied to HTTP, or that the metadata should be independent of the actual datatypes being synchronized. Any given application is going to need to implement not only a synchronization algorithm, but the datatypes being synced. Contrast with git, where the data is standardized (file, directories, commits) and it supports many ways of communicating about the data. (Though, at another level, git doesn't care what's in a file, and that's also true of HTTP.)
- toomim 2y ago
  Yes, we will want to extend URLs to standardize an optional version identifier. It's important for this to be optional. Consider that each keystroke that edits a text resource changes its version ID. You want to be able to link to the text without changing the ID with each keystroke. It's also important to be able to pass versions within headers of a request or response. The server wants to update the version ID with each update it sends to the client, and that's most elegantly done in a header. As for your skepticism about integrating synchronization into HTTP instead of embedding it within a datatype, I can empathize -- but you might be surprised at just how elegantly HTTP extends into a full-featured synchronization protocol. A key to this elegance is the Merge-Type: this is the abstraction that allows a single synchronization algorithm to merge across multiple data types. As an application programmer, you will specify both the data types of your variables (e.g. int, string, bool) and also the merge-types (e.g. "this merges as a bank account balance, or a LWW unique ID, or a collaborative text field"). This is all the application programmer needs to specify. The rest of the synchronization algorithm gets automated by middleware libraries that the programmer can just use and rely upon, like his compiler, and web browser. I'd encourage you to check out the Braid spec, and notice how much we can do with how little. This is because HTTP already has almost everything we need. Compare this with the WebDAV spec, for instance, which tries to define versioning on top of HTTP, and you'll see how monstrous the result becomes. Example here: https://news.ycombinator.com/item?id=40481003 https://news.ycombinator.com/item?id=40481003
ChainOfFools 2y ago
submitted by walterbell nearly all criticisms responded to by a (the?) toomim brother, (who styles himself an independent psychology researcher, interesting) plus an appearance by mike_hearn. Why is this proposal to significantly extend the scope and basic functionality of http so popular with bitcoin promoters?
- walterbell 2y ago
  > submitted by walterbell ... bitcoin promoters Thanks to your left field comment, I've learned from Algolia that my HN submissions and comments include the terms "bitcoin" ~0.3% and "cash" ~1.3%. > nearly all criticisms responded to by a (the?) toomim brother Co-author of the technical proposal? It's usually good to see the author of a technical article providing substantive responses to HN comments. It's also good to have more feedback on the substance of the article. Any comments on the technical proposal? > Why is this proposal to significantly extend the scope and basic functionality of http so popular with bitcoin promoters? This draft proposal could benefit from contribtors from more than one organization (Invisble College), but at least there appear to be multple implementations of the proposed protocol, as is typical of IETF drafts.
  - ChainOfFools 2y ago
    not a front-end developer, so this is not something I'd be directly interacting with although it sounds interesting. But I was struck by the preponderance of questions from other HNers in this comment section, all of which seem to be categorized it under the rubric, "what's the point of this?" It was only then that I noticed the connection between the few people making most of the answers to those questions, as well as the submitter. So I ask again, why are Bitcoiners so clustered around this project?
    - walterbell 2y ago
      > noticed the connection between the few people making most of the answers to those questions, as well as the submitter. As the submitter, what connection are you referencing? Your comment thread is the first I've answered on this article. mike_hearn has one (1) answer. That leaves responses from the author of the article, i.e. one person. Where are the "few people" to whom you are alluding? > questions .. seem to be categorized it under the rubric, "what's the point of this?" Given the effort involved in any IETF draft proposal with at least two competing implementations of the protocol, this is a good question. Since learning about this project 10 hours ago, when the story was submitted, I'm still trying to understand use cases beyond "less complex" than alternatives. Based on the HN responses, it's not a well known project, so it would benefit from wider evaluation and feedback. Work on substantive protocols for P2P decentralized publishing is always welcome, especially in 2024 when LLMs and "AI" are consuming available oxygen. If there is some (unstated?) connection between the proposed protocol and digital payments/assets, then protocol security and access control deserve close attention.
pornel 2y ago
Is it only for realtime? Can it prune history? What if I write a todo application in it, and one of the clients will connect only once a month? What about a client that will drop and never return? (in some systems it means past updates will pile up indefinitely, and state on other clients will keep growing)
- toomim 2y ago
  You can use any CRDT with Braid. Many of them prune history. In fact, we have developed the first pruning p2p text sync algorithm: braid.org/antimatter Indeed, text sync requires old history to merge with old edits. However, Braid does not force you to hold everything. Peers can implement their own policies for deciding who to sync with. They can also ask each other for modules of old history they've forgotten if they realize they need it to merge something. This is made possible with the new Time Machine architecture that we are writing up. This architecture also allows you to sync at multiple time resolutions -- realtime, fine grained, or slower, course grained -- within the same system. Different peers can hold and share at different resolutions. And they can all guarantee consistency with each other in the end. See the recent simpleton algorithm work for an example.
bastawhiz 2y ago
I'm not keen that this extends HTTP instead of building on existing standards. Subscriptions, for instance, could simply be server sent events. Baking braid into the protocol layer itself means it's harder to support it with existing libraries and code. I'm also a little unsure of why "partial PUTs" are needed. "PUT an update as a patch" seems like exactly the use case for PATCH. Not that it super matters either way as far as compatibility is concerned, but it feels like it's contorting PUT to do what PATCH already accomplishes
- toomim 2y ago
  See this comment on SSE: https://news.ycombinator.com/item?id=40482389 https://news.ycombinator.com/item?id=40482389 PATCH is supported in Braid, but only works for client->server. It does not support updates from server->client. So we need a way to express general updates, in both directions. Braid lets you use existing PATCH specs (as Patch-Type) if you want. The PATCH specification also confounds multiple concerns (versioning, conflicts, validation, data formatting, applicable data types) into a single spec. It's nicer for these aspects to be modular and independent. Partial PUT happens to be already a modular, independent way to specify just the patch format, and can be extended to new data types by "Range Units." But use PATCH if you prefer!
  - bastawhiz 2y ago
    I don't feel especially strongly about the PUT vs PATCH distinction. But I left a comment on the thread you linked: I'm just not convinced there's a need for 209. Braid is trying to be both application logic and a transport protocol, but it doesn't have to be. SSE isn't perfect but all of the concerns around it could be addressed with relatively simple solutions.
    - toomim 2y ago
      I'm not convinced on 209, either. There might be a better response code. See the discussion at https://github.com/braid-org/braid-spec/issues/16 https://github.com/braid-org/braid-spec/issues/16 Braid is not trying to be "application logic" or "transport protocol" -- it is a State Synchronization Protocol. If you can think of a more elegant method to support all of state synchronization within SSE, we're all ears! But AFAICS you end up having to embed base64-encoded HTTP responses within SSE events within HTTP responses, which is gross pile of russian dolls, where both the innermost and outermost dolls are the same type of doll -- an HTTP response! So why all the intermediate dolls? It's more elegant and simpler to extend HTTP at the root level, and say: "Instead of returning 1 HTTP response, a Braid server can return N HTTP responses, concatenated together with optional newline separators."
dang 2y ago
Related. Others? Braid: Synchronization for HTTP - https://news.ycombinator.com/item?id=26385480 https://news.ycombinator.com/item?id=26385480 - March 2021 (1 comment) Braid: Synchronization for HTTP - https://news.ycombinator.com/item?id=21626261 https://news.ycombinator.com/item?id=21626261 - Nov 2019 (47 comments)
dagss 2y ago
This allows subscriptions of changes to individual resources... but is more general event streaming a goal? Could it become so? (as in, "give me updates to all resources", not just one). We have been using our own protocol "FeedAPI" for broker-less, multi-service event sourcing over HTTP. But it is a bit home-grown, would be great if event sourcing over HTTP was a more widespread thing with a common standard. There are so many scenarios where event streaming is a great model, but deploying Kafka is overkill, or would couple together the infra too much of publisher and consumer (e.g. different between organizations). https://github.com/vippsas/feedapi-spec https://github.com/vippsas/feedapi-spec
- toomim 2y ago
  Cool! We have been building feeds on braid, too! It would be very interesting to support FeedAPI with Braid -- you would get much faster pushed updates than you currently do with client-pulled updates! See our current work on the "feed" spec in Braidmail: https://braid.org/apps/braidmail/spec2 https://braid.org/apps/braidmail/spec2. To support event-sourcing, you could replace those {link: ...} objects with event objects, and just treat each event as data. Then you'd have a url like /event-stream that peers would subscribe to. Or, you could go deeper, and come up with a custom `Patch-Type:` for your events, which could make your events first-class HTTP citizens that could also be applied to a server using the PATCH method. Then your url might be /system-state, and represent the state for the whole system. Braid supports such custom events, but be aware that this model is limited in a few ways: - It's less interoperable, because all peers must know and implement the custom event types (aka patch-type) in order to read or write state. - You don't get to re-use the beautiful and complicated merge algorithms that support eventual consistency with multiple writers (the OT/CRDT algorithms). - It doesn't allow summarization of history. It's hard to prune history when each peer needs to know all events in order to do anything. You might also be interested in PREP, which is a pure event-update protocol being worked on by my colleague Rahul Gupta: https://github.com/CxRes/prep https://github.com/CxRes/prep We're trying to merge our work together, if we can. You might also be interested in our deliberation on the pros/cons of "events" vs. "state updates" here: https://github.com/braid-org/braid-spec/issues/102#issuecomment-1770468174 https://github.com/braid-org/braid-spec/issues/102#issuecomm...
walterbell 2y ago
A handful of iOS apps (2Do, Omnifocus, DevonThink, PhotoSync, GoodReader) support user-hosted WebDAV/CalDAV storage for state sync between devices. If Braid can lower the cost for apps to synchronize state across devices without "the cloud", that would be a win for decentralized infrastructure.

233 points by walterbell 2y ago

victorbjorklund 2y ago

I dont get it. What would I use this for? (Honest question. Im probably not smart enough to get it)

chronicler 2y ago
same reason you'd use HTTP, it's a superset of HTTP
- toomim 2y ago
  Yes, and to elaborate — for any HTTP usage with dynamic state. HTTP was invented for static pages, that were written by hand, and rarely changed. But today's web has dynamic pages, driven by javascript and databases, and users expect them to update in realtime. Braid-HTTP adds support for dynamic state to HTTP.
- victorbjorklund 2y ago
  I still dont get it. Is this to sync two servers between each other?
  - whilenot-dev 2y ago
    From the article: > Together, these features enable a web resource to synchronize automatically across multiple clients, servers and proxies, and support arbitrary simultaneous edits by multiple writers, under arbitrary network delays and partitions, while guaranteeing consistency using an OT, CRDT, or other algorithm. So, you'd use it if you want to "support arbitrary simultaneous edits by multiple writers, under arbitrary network delays and partitions, while guaranteeing consistency", for example for apps that want to support collaborative editing. Braid wants to turn state transfer (like in RESTful API designs - Representational State Transfer) into a kind of synchronization transfer for state. Currently, state synchronization is handled on the state management layer of each individual application, and API calls just transfer that state, but Braid wants to have it solved as an extension to HTTP libraries on the API layer of an application AFAICS.
    - toomim 2y ago
      Yes, or even more simply: Let's say that your client runs GET /some-json, and that JSON gets updated, and the client wants to get the updates. Right now, your options are: 1. Re-run the GET /some-json again from the client (polling) 2. Start a websocket, and invent some custom protocol over the websocket to let the client subscribe to /some-json. With Braid, you just do: GET /some-json Subscribe: true And the client will automatically get the updates, within HTTP. The Braid-HTTP ponyfill library abstracts the details behind a normal fetch() for you, until browsers implement it. Today, we tend to use HTTP for static assets, and then switch to a websocket or SSE or something whenever things get dynamic. That sucks! We lose the standard of HTTP! Braid lets you keep using HTTP, even for your dynamic assets. It also solves issues in caching. Instead of using max-age heuristics, we have actual real versioning! And yes— the protocol generalizes all the way to full-peer-to-peer multi-writer CRDT/OT algorithms. But start simple!
      - whilenot-dev 2y ago
        Frankly, your explanation isn't any simpler, quite the contrary. > That sucks! We lose the standard of HTTP! Why does this suck? WebSockets or SSEs are also standardized. > the protocol generalizes all the way to full-peer-to-peer multi-writer Why "peer-to-peer"? While Braid can be used in a local-first setting and probably do peer-to-peer, that isn't the focus here. The focus is to help application developers to not get into the shenanigans of CRDTs and distributed algorithms etc., but solve those things already via polyfills.
        toomim 2y ago
        WebSockets and SSE are standards in the same way that TCP is a standard -- if you use them, you are still defining an ad-hoc protocol on top to subscribe to state and publish new state. HTTP is a standard on top of TCP. It provides a higher-level abstraction than a socket -- the abstraction of State Transfer. When you use a WebSocket, you're back to a low-level socket, and have to redefine "state", and the methods to get it, put it, and subscribe to it. Since each web programmer defines those methods in a different way, his state gets hidden behind his own non-standard protocol. There is no way for website A to re-use the state on website B, for instance, without learning website B's custom WebSocket protocol and re-implementing it perfectly. CDNs and other caches cannot handle WebSocket traffic. But if you use a standard like Braid-HTTP, they can cache your dynamic assets along with your static assets.
  - toomim 2y ago
    You could sync two servers. Or you could sync a server with clients. Braid is transitioning HTTP into a peer-to-peer world, where the distinction between clients and servers doesn't matter.
    - victorbjorklund 2y ago
      Could you call it websockets without websockets?
      - toomim 2y ago
        In some sense, you could! The reason is that almost every use WebSockets is actually for Synchronizing State. What you really want to do is to synchronize state. Braid is a protocol for doing just that. So you don't need to turn to WebSockets anymore!
        victorbjorklund 2y ago
        I wonder if Braid could be a transport layer for Phoenix Liveview. It uses Websockets but falls back to long-polling if websockets isnt available.
        toomim 2y ago
        Yes! Phoenix Liveview is very similar to the Statebus project (https://stateb.us https://stateb.us) that inspired Braid. Braid was the protocol that Statebus needed, and Phoneix Liveview is one of the most exciting projects in the Statebus space that I know! I think we are working from a common inspiration!
laminatedsmore 2y ago
I didn't get this either until I looked at the linked doc, which breaks it down more obviously imo (section 6.1 for examples). Looks interesting to me, seems nice to be able to handle refreshing content with pure http as described here rather than with web sockets. https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http#section-6.1 https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
sesm 2y ago
From what I understood, it's a protocol for exposing a tree of files for synchronization and concurrent editing, similar to WebDAV.

bawolff 2y ago

I cant help but think that this feels like webDav on steriods, and webdav never really caught on. I kind of think sometimes it makes more sense to layer on top of http instead of extending.

mike_hearn 2y ago
It kind of is layered on top, no? WebDAV didn't really catch on, but the general product space of remote drives did. The problem with WebDAV is that like most Web* tech (and maybe Braid) it's design by committee in the abstract. To make remote drives work well for the end user requires a fairly complicated protocol with tons of ugly edge cases. The interoperable standards from committees approach tends to fail in those situations, whereas hard driving startups that use proprietary protocols they can iterate quickly tend to win. Iteration speed > openness, when there's nothing to copy from. Standardization tends to be about bigger companies trying to commodify their competitors.
- 0x006A 2y ago
  many proprietary protocols iterate themself out of existence, just because some of them survive does not mean its the better model. Look at the video codec space, proprietary codes have long lost and have gone the way of Real Media, Windows Media Video and many others. While H.26X which is the result of design by committee are used everywhere.
  - treyd 2y ago
    It might be a different scenario with (modern) codecs where to get good use out of them it helps a lot to ship dedicated hardware. That's not true of file sync protocols.
- giovannibonetti 2y ago
  > Standardization tends to be about bigger companies trying to commodify their competitors. Didn't you mean to commodity their complements? Like when Microsoft (software company) commoditied the hardware on top of where their OS ran.

dotancohen 2y ago

Interesting idea, I would certainly have use for it. But the current severside library support is limited to Lua and Haskell - quite the sign that the project is squarely in the hands of academics and has never seen real world dependent on a heavily used system. Even if my stack were based on a supported language, I would take pause at that.

lionkor 2y ago
luajit is stupid fast, and cooperating with a lua program from other languages is trivial. Might be worth a look?
- dotancohen 2y ago
  It is certainly worth a look, and of course if the project takes off then Python and other library support would be expected eventually. That's how open source communities usually develop. My point was that the current library support is a good proxy for the state of development and deployment, and currently this state is very early.
  - mst 2y ago
    Given there's both client and server code available for JS that seems like a perfectly reasonable place to start experimenting with it. The python community tends to be less neophilic than JS (often to its advantage) so I wouldn't expect that to show up until later.
josephg 2y ago
There are a couple nodejs implementations as well. I would know - I wrote one of them.
- junto 2y ago
  This looks somewhat similar to websockets and SignalR. What am I missing here? Is it like GRPC as a protocol, built on top of HTTP/2? I’d be interested in working on a port for .NET.

BigParm 2y ago

Can someone ELIF? I can synchronize clients around some state using http. What is an http extension? Does it add new http methods? If so, will anyone else be able to respond to these methods? Isn't http stateless what do you mean versioning? And etcetera! I'm sorry to criticize but who is this article written for? It's either so dense that you already need to be an expert to unpack all the information, or the information is not there. I would recommend starting from the assumption that the Hacker News layman knows the http protocol. Make sure what you show us is possible to follow from some reasonable starting point.

toomim 2y ago
HTTP (HyperText Transfer Protocol) is a State Transfer protocol. ReST (Representational State Transfer) is a State Transfer architecture. It was built for transferring a page from server to client. But if the page changed afterward, the protocol threw up its hands, and left it to the user to click "reload." Braid proposes new HTTP headers that give HTTP new features, so that instead of just being a State Transfer protocol, it gains the functionality to become a full State Synchronization protocol. To "extend" a protocol means to add new features to it, e.g. with new headers, which doesn't break existing uses of the protocol, but allows new implementations to opt into new extended functionality. HTTP was originally stateless, but then added cookies. Over time, web apps evolved to store tons of state in databases on the server. Then they started storing tons of state in javascript variables and DOM on the client. Today, most of the code we write in a web app is there to synchronize the changing state on a server with the changing DOM on the client. The web is no longer static pages. It is dynamic state. But since we haven't extended HTTP to support dynamic state, programmers have to write all this synchronization code by hand. This has been a pain in the ass, and so we've evolved monstrous stacks of Javascript frameworks (react, redux, etc. etc.) to try to help manage this state synchronization from server to client. If we build this functionality directly into HTTP, tons of code goes away, and the web itself gains great new features. Then, since we will have a standard for state synchronization, our state can interoperate, and walled siloes of websites break down. We can have a P2P web. When this work is complete, the browser will no longer need a reload button. The browser/server will guarantee that every page is always up-to-date, and every peer that synchronizes with that state will have an equal copy.

toomim 2y ago

We're about to release a new braid-text library: https://github.com/braid-org/braid-text/ https://www.npmjs.com/package/braid-text This is the easiest way to add collaborative editing to a web app. You can add it to any (req, res) handler in your nodejs app. No websocket necessary, since it extends HTTP itself! Plus, it features a new `simpleton` merge-type that requires zero history overhead on the client, and you can implement the client protocol from scratch in just 50 lines of code. There's no reason not to add collaborative editing to every user-editable string in your app now! All the network traffic is just HTTP, with an open, easy-to-read protocol that supports any OT or CRDT (currently defaulting to josephg's Diamond-Types). You can even read and write directly using the Braid-Chrome extension, which adds a devtools panel to view the version history of any Braid-HTTP resource: https://github.com/braid-org/braid-chrome https://github.com/braid-org/braid-chrome. I'm really excited about this library. We were going to announce it tomorrow (at https://braid.org/meeting-86 https://braid.org/meeting-86), but since this hacker news discussion is happening... I can't hold myself back from talking about it now!

notpushkin 2y ago
Neat! Any plans to release a Firefox extension, too? It shouldn't be too hard, IIRC the API is largely compatible.
- toomim 2y ago
  That is cool! Nobody has worked on a FireFox devtool yet, but we welcome contributions.
remram 2y ago
> No websocket necessary, since it extends HTTP itself! What does that mean? How do other clients get notified of remote changes? edit: Braid include its own SSE-like construct, completely incompatible, with a brand-new HTTP status code "209". Seems unnecessary... How does that work with existing servers/proxies/middleboxes?
- toomim 2y ago

halayli 2y ago

HTTP is a request-response protocol. It doesn't deal with state transfer nor does it define it. I find it weird that the first thing they mention is this: > Braid-HTTP is an extension to HTTP that generalizes it from a state transfer to a state synchronization protocol. They could simply say it's a synchronization protocol over HTTP.

toomim 2y ago
> HTTP ... doesn't deal with state transfer nor does it define it. The HTTP acronym stands for HyperText Transfer Protocol. ReST stands for Representational State Transfer. HTTP began as a HTML Transfer Protocol, but then generalized to content-types beyond HTML, such as images, scripts -- and general state. So today HTTP is known as a State Transfer Protocol. This architecture was canonicalized as ReST in Roy Fielding's dissertation. In order to generalize HTTP from State Transfer to State Synchronization, we have to augment some parts of it, such as request/response. Instead of just getting a single response to a request, Braid allows a client to subscribe to a resource, with a single request which will be then given multiple responses -- one response for each update to the resource. It is quite elegant to extend HTTP at this level! And we can do so without requiring any changes to web browsers.

EternalFury 2y ago

I wonder how well that scales and how it handles network partitions or erratic peers.

toomim 2y ago
You can use CRDT algorithms with Braid that heal from network partitions transparently, and guarantee strong eventual consistency. Here's an example CRDT algorithm for collaborative text that can handle arbitrary network partitions, and guarantee full eventual consistency, and prune all unnecessary history: https://braid.org/antimatter https://braid.org/antimatter This algorithm is the first to do so, and happened to be developed in the Braid group. As for scaling, check out JosephG's https://josephg.com/blog/crdts-go-brrr/ https://josephg.com/blog/crdts-go-brrr/ CRDT, which scales great, and is thoroughly tested. The Braid protocol is also architected with a new type of OT/CRDT architecture called a Time Machine that lets you do advanced things like apply backpressure through a network to decrease the frequency of updates, which we presented in https://braid.org/meeting-81 https://braid.org/meeting-81, and are releasing in the https://github.com/braid-org/braid-text https://github.com/braid-org/braid-text library that you can try right now.

superkuh 2y ago

For an HTTP company they sure do hate hypertext. Their entire write-up is just a blank white page unless one successfully executes the javascript from 5 separate domains.

toomim 2y ago
Point taken. But to be fair, we're not a company, but a working group. And we don't hate hypertext, we just haven't gotten around to implementing server-side rendering, which would be a nice thing to do.
- superkuh 2y ago
  Glad to hear it. What on that page of just text and images made you feel an entire application was appropriate instead of just a hypertext document? I hope this application-centric approach isn't also being applied to the HTTP extension. Hypertext documents should at least get an equal share of the consideration for an HTTP protocol. I get that for commerce HTTP is just a transport to deliver the javascript/json/etc application. That's the way things are. But HTTP in general has to handle more use cases than just commerce. I worry that adding this dynamic checking of state to HTTP itself will make websites that adopt it inaccessible and unusable for almost every browser that currently exists. Only new software will be able to use them. It's a substantial break. Maybe don't call it HTTP.

skybrian 2y ago

They seem to have made a choice that URL's do not include version numbers. Instead, the version is sent as a separate header. That would make it hard to link to a specific version. I suppose there's nothing stopping anyone from also having URL's for specific versions, but it's not standardized. More generally, I'm unconvinced that synchronization should be so closely tied to HTTP, or that the metadata should be independent of the actual datatypes being synchronized. Any given application is going to need to implement not only a synchronization algorithm, but the datatypes being synced. Contrast with git, where the data is standardized (file, directories, commits) and it supports many ways of communicating about the data. (Though, at another level, git doesn't care what's in a file, and that's also true of HTTP.)

toomim 2y ago
Yes, we will want to extend URLs to standardize an optional version identifier. It's important for this to be optional. Consider that each keystroke that edits a text resource changes its version ID. You want to be able to link to the text without changing the ID with each keystroke. It's also important to be able to pass versions within headers of a request or response. The server wants to update the version ID with each update it sends to the client, and that's most elegantly done in a header. As for your skepticism about integrating synchronization into HTTP instead of embedding it within a datatype, I can empathize -- but you might be surprised at just how elegantly HTTP extends into a full-featured synchronization protocol. A key to this elegance is the Merge-Type: this is the abstraction that allows a single synchronization algorithm to merge across multiple data types. As an application programmer, you will specify both the data types of your variables (e.g. int, string, bool) and also the merge-types (e.g. "this merges as a bank account balance, or a LWW unique ID, or a collaborative text field"). This is all the application programmer needs to specify. The rest of the synchronization algorithm gets automated by middleware libraries that the programmer can just use and rely upon, like his compiler, and web browser. I'd encourage you to check out the Braid spec, and notice how much we can do with how little. This is because HTTP already has almost everything we need. Compare this with the WebDAV spec, for instance, which tries to define versioning on top of HTTP, and you'll see how monstrous the result becomes. Example here: https://news.ycombinator.com/item?id=40481003 https://news.ycombinator.com/item?id=40481003

ChainOfFools 2y ago

submitted by walterbell nearly all criticisms responded to by a (the?) toomim brother, (who styles himself an independent psychology researcher, interesting) plus an appearance by mike_hearn. Why is this proposal to significantly extend the scope and basic functionality of http so popular with bitcoin promoters?

walterbell 2y ago
> submitted by walterbell ... bitcoin promoters Thanks to your left field comment, I've learned from Algolia that my HN submissions and comments include the terms "bitcoin" ~0.3% and "cash" ~1.3%. > nearly all criticisms responded to by a (the?) toomim brother Co-author of the technical proposal? It's usually good to see the author of a technical article providing substantive responses to HN comments. It's also good to have more feedback on the substance of the article. Any comments on the technical proposal? > Why is this proposal to significantly extend the scope and basic functionality of http so popular with bitcoin promoters? This draft proposal could benefit from contribtors from more than one organization (Invisble College), but at least there appear to be multple implementations of the proposed protocol, as is typical of IETF drafts.
- ChainOfFools 2y ago
  not a front-end developer, so this is not something I'd be directly interacting with although it sounds interesting. But I was struck by the preponderance of questions from other HNers in this comment section, all of which seem to be categorized it under the rubric, "what's the point of this?" It was only then that I noticed the connection between the few people making most of the answers to those questions, as well as the submitter. So I ask again, why are Bitcoiners so clustered around this project?
  - walterbell 2y ago
    > noticed the connection between the few people making most of the answers to those questions, as well as the submitter. As the submitter, what connection are you referencing? Your comment thread is the first I've answered on this article. mike_hearn has one (1) answer. That leaves responses from the author of the article, i.e. one person. Where are the "few people" to whom you are alluding? > questions .. seem to be categorized it under the rubric, "what's the point of this?" Given the effort involved in any IETF draft proposal with at least two competing implementations of the protocol, this is a good question. Since learning about this project 10 hours ago, when the story was submitted, I'm still trying to understand use cases beyond "less complex" than alternatives. Based on the HN responses, it's not a well known project, so it would benefit from wider evaluation and feedback. Work on substantive protocols for P2P decentralized publishing is always welcome, especially in 2024 when LLMs and "AI" are consuming available oxygen. If there is some (unstated?) connection between the proposed protocol and digital payments/assets, then protocol security and access control deserve close attention.

pornel 2y ago

Is it only for realtime? Can it prune history? What if I write a todo application in it, and one of the clients will connect only once a month? What about a client that will drop and never return? (in some systems it means past updates will pile up indefinitely, and state on other clients will keep growing)

toomim 2y ago
You can use any CRDT with Braid. Many of them prune history. In fact, we have developed the first pruning p2p text sync algorithm: braid.org/antimatter Indeed, text sync requires old history to merge with old edits. However, Braid does not force you to hold everything. Peers can implement their own policies for deciding who to sync with. They can also ask each other for modules of old history they've forgotten if they realize they need it to merge something. This is made possible with the new Time Machine architecture that we are writing up. This architecture also allows you to sync at multiple time resolutions -- realtime, fine grained, or slower, course grained -- within the same system. Different peers can hold and share at different resolutions. And they can all guarantee consistency with each other in the end. See the recent simpleton algorithm work for an example.

bastawhiz 2y ago

I'm not keen that this extends HTTP instead of building on existing standards. Subscriptions, for instance, could simply be server sent events. Baking braid into the protocol layer itself means it's harder to support it with existing libraries and code. I'm also a little unsure of why "partial PUTs" are needed. "PUT an update as a patch" seems like exactly the use case for PATCH. Not that it super matters either way as far as compatibility is concerned, but it feels like it's contorting PUT to do what PATCH already accomplishes

toomim 2y ago
See this comment on SSE: https://news.ycombinator.com/item?id=40482389 https://news.ycombinator.com/item?id=40482389 PATCH is supported in Braid, but only works for client->server. It does not support updates from server->client. So we need a way to express general updates, in both directions. Braid lets you use existing PATCH specs (as Patch-Type) if you want. The PATCH specification also confounds multiple concerns (versioning, conflicts, validation, data formatting, applicable data types) into a single spec. It's nicer for these aspects to be modular and independent. Partial PUT happens to be already a modular, independent way to specify just the patch format, and can be extended to new data types by "Range Units." But use PATCH if you prefer!
- bastawhiz 2y ago
  I don't feel especially strongly about the PUT vs PATCH distinction. But I left a comment on the thread you linked: I'm just not convinced there's a need for 209. Braid is trying to be both application logic and a transport protocol, but it doesn't have to be. SSE isn't perfect but all of the concerns around it could be addressed with relatively simple solutions.
  - toomim 2y ago
    I'm not convinced on 209, either. There might be a better response code. See the discussion at https://github.com/braid-org/braid-spec/issues/16 https://github.com/braid-org/braid-spec/issues/16 Braid is not trying to be "application logic" or "transport protocol" -- it is a State Synchronization Protocol. If you can think of a more elegant method to support all of state synchronization within SSE, we're all ears! But AFAICS you end up having to embed base64-encoded HTTP responses within SSE events within HTTP responses, which is gross pile of russian dolls, where both the innermost and outermost dolls are the same type of doll -- an HTTP response! So why all the intermediate dolls? It's more elegant and simpler to extend HTTP at the root level, and say: "Instead of returning 1 HTTP response, a Braid server can return N HTTP responses, concatenated together with optional newline separators."

dang 2y ago

Related. Others? Braid: Synchronization for HTTP - https://news.ycombinator.com/item?id=26385480 https://news.ycombinator.com/item?id=26385480 - March 2021 (1 comment) Braid: Synchronization for HTTP - https://news.ycombinator.com/item?id=21626261 https://news.ycombinator.com/item?id=21626261 - Nov 2019 (47 comments)

dagss 2y ago

This allows subscriptions of changes to individual resources... but is more general event streaming a goal? Could it become so? (as in, "give me updates to all resources", not just one). We have been using our own protocol "FeedAPI" for broker-less, multi-service event sourcing over HTTP. But it is a bit home-grown, would be great if event sourcing over HTTP was a more widespread thing with a common standard. There are so many scenarios where event streaming is a great model, but deploying Kafka is overkill, or would couple together the infra too much of publisher and consumer (e.g. different between organizations). https://github.com/vippsas/feedapi-spec https://github.com/vippsas/feedapi-spec

toomim 2y ago
Cool! We have been building feeds on braid, too! It would be very interesting to support FeedAPI with Braid -- you would get much faster pushed updates than you currently do with client-pulled updates! See our current work on the "feed" spec in Braidmail: https://braid.org/apps/braidmail/spec2 https://braid.org/apps/braidmail/spec2. To support event-sourcing, you could replace those {link: ...} objects with event objects, and just treat each event as data. Then you'd have a url like /event-stream that peers would subscribe to. Or, you could go deeper, and come up with a custom `Patch-Type:` for your events, which could make your events first-class HTTP citizens that could also be applied to a server using the PATCH method. Then your url might be /system-state, and represent the state for the whole system. Braid supports such custom events, but be aware that this model is limited in a few ways: - It's less interoperable, because all peers must know and implement the custom event types (aka patch-type) in order to read or write state. - You don't get to re-use the beautiful and complicated merge algorithms that support eventual consistency with multiple writers (the OT/CRDT algorithms). - It doesn't allow summarization of history. It's hard to prune history when each peer needs to know all events in order to do anything. You might also be interested in PREP, which is a pure event-update protocol being worked on by my colleague Rahul Gupta: https://github.com/CxRes/prep https://github.com/CxRes/prep We're trying to merge our work together, if we can. You might also be interested in our deliberation on the pros/cons of "events" vs. "state updates" here: https://github.com/braid-org/braid-spec/issues/102#issuecomment-1770468174 https://github.com/braid-org/braid-spec/issues/102#issuecomm...

walterbell 2y ago

A handful of iOS apps (2Do, Omnifocus, DevonThink, PhotoSync, GoodReader) support user-hosted WebDAV/CalDAV storage for state sync between devices. If Braid can lower the cost for apps to synchronize state across devices without "the cloud", that would be a win for decentralized infrastructure.