10 ms·

HTTP Feeds

214 points by mcp_ 5y ago

talideon 5y ago
Way back I recall (ab)using chunked encoding to do something similar. Now there's an underexploited element of HTTP...
raggi 5y ago
JSON inside server sent events gives you a browser native client, no need for streaming extensions to your JSON lib which is kinda rare, and various intermediates know not to buffer the content type already. Sorry to be the what about X guy, maybe there's something I missed?
- tehbeard 5y ago
  Yeah this feels very much like someone retyped the Server Sent Events spec and bolted an arbitrary spec on for each message...
  - jpgvm 5y ago
    Yeah batched arrays isn't the way to go when when you want to mimic what you would get from a traditional message broker. This really should be a spec that builds on SSE but specifies the encoding (CloudEvents) and parameters to negotiate the serialisation (JSON/Protobuf/Avro) and lastEventId, then it would be much more useful.
sparsely 5y ago
This seems like a really attractive way of publishing events to a third party. Platform independent and easy to understand.
- dotancohen 5y ago
  > Platform independent and easy to understand. The platform independence is implicit in the "HTTP" part of the name.
technicolorwhat 5y ago
The idea is nice and needed. However maybe the spec is a bit elaborate for me to adopt it immediately. I've been rolling my own for some time at some clients, for our kafkaesque/event sourcing patterns. However what I used there was simple http stream/json stream like this: - No start of [] but JSON newline entries a new line is an new entry - Using Anything as an id (we've been using redis XSTREAMS as lightweight kafka concepts, just 64bit integers) - have an type as an event, and versioning is just done by upgrading the type, ugly, but easy. - We'er considering using SSE at this moment Compaction is not something that I would do in the protocol I think I would just expose another version of it on a different url I think or put it in a different spec.
- xg15 5y ago
  How do you recover when there is e.g. some connection error and a client misses some events? Can the client ask to replay the events? Then, how far back can they go?
lexicality 5y ago
Feels like a bit silly not to have made this compatible with SSE since all the mechanisms for acessing that are built into browsers these days. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events https://html.spec.whatwg.org/multipage/server-sent-events.ht...
- technicolorwhat 5y ago
  Yeah, we're also pushing for SSE for this kind of things.
  - lexicality 5y ago
    Having a standard for what events look like is a decent idea as people generally just make some random thing up without thinking too hard and having a standard is generally helpful - but making your own polling system etc when there's already a battle tested one available seems like wasted effort and a bit counterproductive
  - horstmeyer 5y ago
    Does SSE work reliably in mobile browsers nowadays? A few years back when I tried, it was working OK in Chrome for Android but mobile browser support didn't seem complete.
    - gunapologist99 5y ago
      Sadly, there are very few mobile browsers that aren't Chrome or Safari. In fact, Safari/Webkit is the only mobile browser engine available on IOS at all -- all other browsers are just chrome (pun intended) around IOS webkit. To your point, https://caniuse.com/eventsource https://caniuse.com/eventsource Also there are polyfills for very old browsers like IE.
  - mromanuk 5y ago
    I had previously used SSE in a project and had no idea it existed or that it was widely supported by most browsers. It's a beautiful and straightforward technology, but it doesn't seem to be as widely used as it should be, and more tooling is required. Also more tooling is required.
outsomnia 5y ago
How about "catching up" on events, if a client had an outage, can it indicate the last event ID it had and then get a natural replay of the older events?
- tirpen 5y ago
  That's the second example on the page.
  - outsomnia 5y ago
    Doh... thanks for pointing it out. I asked because it says elsewhere that persistence of objects is a non-goal, I guess it meant storing them clientside or something.
deleted 5y ago
[deleted]
chaz6 5y ago
My first thought is if the client does not specify a start ID it could be sent billions of records depending on the elapsed time and frequency of events. What would be the best way to avoid overloading a client?
- quaintdev 5y ago
  I think the server will give a truncated response. They need to clearly document what is the behaviour for this case.
brasetvik 5y ago
It'd be great to get a fairly standard way of doing this. :) Having worked in this problem space a bit recently, I find this part a bit too optimistic: > The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events. The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer. Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though. Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences. The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup) A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.
- 0des 5y ago
  Perhaps the time problem can be handled with an eventually consistent Lamport clock system
  - jpgvm 5y ago
    Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes. Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.
josephg 5y ago
Very cool! This looks very similar to what we're doing with the braid spec[1], though I really like how clear and concise your examples are! This is a great little website. Some differences between our approaches: - I think its a good idea to support arbitrary patch formats via a content-type style field, just like we have different formats for images. This lets you use the same protocol for things like collaborative editors. - For some data sets (like CRDTs), you want each change to be able to refer to multiple "parents". - Your protocol is very JSON-y and not very HTTP-y. It looks like you're basically using JSON to express HTTP. Why not just use HTTP? One big downside of the JSON approach is that it makes it awkward to transmit binary patches. (Eg, 'patching' an image) Feel free to reach out if you're up for a chat! Looks like we're working on the same problem. [1] https://github.com/braid-org/braid-spec/blob/master/draft-toomim-httpbis-braid-http-03.txt https://github.com/braid-org/braid-spec/blob/master/draft-to...
- 0des 5y ago
  This made me all warm and fuzzy inside that nerds can still help other nerds even when we are on different teams. Hope y'all are able to share some ideas.
rendall 5y ago
This could form the foundation of a distributed social media network.
catwell 5y ago
I wrote a blog post advocating for something like this recently, with an additional push to notify the client when it should poll: https://blog.separateconcerns.com/2022-03-05-push-to-poll.html https://blog.separateconcerns.com/2022-03-05-push-to-poll.ht... I wasn't aware of this but it fits the use case perfectly, I will update the post.
- jkarneges 5y ago
  Indeed, these kinds of abbreviated notifications are a great way to implement realtime updates. The code is simpler and there's less chance of screwing something up. I've also seen this described as hints: https://blog.andyet.com/2014/09/23/send-hints-not-data/ https://blog.andyet.com/2014/09/23/send-hints-not-data/
wrren 5y ago
I like this idea, however I think the event ID being encoded in the response body places constraints on what each element looks like. Perhaps it would make more sense to encode the last ID/feed position in a response header and have the client submit that in a subsequent request header? That would decouple the feed position from any one element or the response structure itself.
sdze 5y ago
I always implement my own standards. I am faster that way instead of learning what others created. Obviously this is only true for less complex subjects.
- adenozine 5y ago
  See also: technical debt
  - fuckyouwedodata 5y ago
    Not if you keep using your standards across different projects
fuckyouwedodata 5y ago
Is it sensible to keep the TCP connection alive for long periods of time, reading in events as ljson? I imagine that this is no issue for QUIC
- jon_adler 5y ago
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
  - fuckyouwedodata 5y ago
- Karrot_Kream 5y ago
  Yeah there's no problems with a long-lived TCP connection unless one end of the connection tries to drop the connection after a certain amount of time (which certain firewalls do.)
  - fuckyouwedodata 5y ago
    Sounds good. Thanks for pointing out the issue with some firewalls
pfsalter 5y ago
This seems similar, but not as fleshed out as RPDE feeds, which has already been accepted by the W3 [1], and provides many answers to lots of the 'eventually up to date' questions. [1] https://www.w3.org/2017/08/realtime-paged-data-exchange/ https://www.w3.org/2017/08/realtime-paged-data-exchange/
rmetzler 5y ago
No pagination?
- Fiahil 5y ago
  You probably mean "maximum batch size", because pagination is handled by the lastEventId and the fact that a stream is always consumed in order.
- aiobe 5y ago
  Correct. No fixed sized pages, but dynamic batches based on the lastEventId. This is much easier to implement, both in server and client side, and it greatly removed the amount of data transferred. With fixed pages you would return content of the latest page for every poll request until it is "full".
xg15 5y ago
It's an interesting idea, but I wonder how well the promise will hold up that this is less complex than using a message broker. Yes, the network topology and protocol are certainly less complex, but there are now additional strong requirements how an endpoint has to store and manage existing events. (See the bits about strict time ordering, compaction, aggregation, etc). A lot of this is effectively what a message broker is doing in a "traditional" system to guarantee consistency. Those tasks aren't gone, they are just pushed to the endpoints now.
- efunneko 5y ago
  My thoughts exactly. While this is more human readable on the wire, a message broker delivering the feed would provide many different other features that might be useful, such as transactions, load-balancing, guaranteed delivery and per-endpoint state to simplify the individual application instances. For those not aware of what message brokers are, there are many to choose from such as: Mosquito, RabbitMQ, ActiveMQ, Solace... If delivery over HTTP is a requirement, many of these brokers support delivery over websockets or (in the case of Solace) also support long polling.
- aiobe 5y ago
  A few issues with message brokers, esp. in the system-to-system integration: - Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... - Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, ... Who is on call when something goes wrong? - Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
  - xg15 5y ago
    > Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... That's a valid point. I think it's a pity we don't have an equivalent standard for asynchronous messaging with the same support as HTTP. However, there are lots of options for presenting an asynchronous public API that would use your message broker behind the scenes, without fully exposing it: Websockets, SSE, web hooks, etc... > Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, True, but don't you have that anyway? How is this different from requesting a new database, service route, service definition, etc? > Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting... How simple or complex this is depends on the concrete broker at hand. There are some protocols, e.g. STOMP that are simple enough that you could write your own client. And as I wrote in the parent: HTTP feeds are a technology as well. You'll have to think about troubleshooting and resiliency there as well.
xg15 5y ago
With more attention going to long polling again, I wonder if it would be useful to introduce some kind of HTTP signaling (header or 1xx status) to indicate on the protocol level that long polling is going on. This might be useful information for intermediaries - e.g. proxies, firewalls, browser network tab, etc.
yashasolutions 5y ago
This looks great. Has anyone any insights or reference about the limits of this approach compared to more traditional message queues?
- dgritsko 5y ago
  One obvious downside is that polling at a fixed interval is going to be less efficient than having an event "pushed" to your client as soon as it's available. If your events are low-volume, then there will be periods where your polling requests return no new data, but you still have to make those requests anyway in order to determine that. And conversely, if your events are high-volume, then your polling interval represents an arbitrary amount of delay that you're introducing into the system. That might not be big deal for some applications, but it's worth mentioning as a potential downside in situations where you want the behavior to be as near-instantaneous as possible.
diordiderot 5y ago
Can someone ELI5 the significance of this? * What are some scenarios where you would need a feed? * What's being done now to solve the problem and how is this different? Many thanks
- kitd 5y ago
  This basically puts a REST/HTTP GET frontend on your messaging backend (Kafka, RabbitMQ, MQTT, etc). So whatever events you want to expose publicly can be done over plain HTTP. Having said that, IMO using SSE or websockets would be a better fit than raw HTTP.
- tiernano 5y ago
  I could see this being used as part of a distributed system for shopping. for example, say you have 20 stores, each with its own storage area for items, but you also have 2 large warehouses. Your online site might need to know which stores have stock now, or which warehouses have stock. The site might allow you to do "Click and collect" do it would need to know, in somewhat real time, which stores actually have the stock. each store would have its own endpoint with a data feed that the central server can get data from. Same with online orders for delivery. It needs to know what warehouses have the stock, and if none do, how to get the store to ship it to a customer. Likewise, the stores might need to know what is in the warehouses, or even across town; someone walks in store to order something, but its not in that store. But they know, in real time, that its in the warehouse for delivery next day or that the store across the city has it. What is done now is a more centralised approach; all sites would have a connection to a single DB in head office that stores everything. This makes things more distributed and, in theory, removes a single point of failure. I should clarify, I am not working on any of this, this is just how I think it would work... if anyone wants to step in and tell me if I'm wrong, right, or just plan stupid, please shout.
  - diordiderot 5y ago
    Great answer and exactly what I was looking for! Thank you
yencabulator 5y ago
I think serving data from multiple Kafka partitions gets unnecessarily hard if your "continue from this point" token is tied to a singular event ID. For that, it'd be better to have the "cursor token" be an arbitrary blob of data you repeat back to the server. Then the server can e.g. encode a list of partition->offset values into it.

214 points by mcp_ 5y ago

Way back I recall (ab)using chunked encoding to do something similar. Now there's an underexploited element of HTTP...

JSON inside server sent events gives you a browser native client, no need for streaming extensions to your JSON lib which is kinda rare, and various intermediates know not to buffer the content type already. Sorry to be the what about X guy, maybe there's something I missed?

tehbeard 5y ago
Yeah this feels very much like someone retyped the Server Sent Events spec and bolted an arbitrary spec on for each message...
- jpgvm 5y ago
  Yeah batched arrays isn't the way to go when when you want to mimic what you would get from a traditional message broker. This really should be a spec that builds on SSE but specifies the encoding (CloudEvents) and parameters to negotiate the serialisation (JSON/Protobuf/Avro) and lastEventId, then it would be much more useful.

sparsely 5y ago

This seems like a really attractive way of publishing events to a third party. Platform independent and easy to understand.

dotancohen 5y ago
> Platform independent and easy to understand. The platform independence is implicit in the "HTTP" part of the name.

technicolorwhat 5y ago

The idea is nice and needed. However maybe the spec is a bit elaborate for me to adopt it immediately. I've been rolling my own for some time at some clients, for our kafkaesque/event sourcing patterns. However what I used there was simple http stream/json stream like this: - No start of [] but JSON newline entries a new line is an new entry - Using Anything as an id (we've been using redis XSTREAMS as lightweight kafka concepts, just 64bit integers) - have an type as an event, and versioning is just done by upgrading the type, ugly, but easy. - We'er considering using SSE at this moment Compaction is not something that I would do in the protocol I think I would just expose another version of it on a different url I think or put it in a different spec.

xg15 5y ago
How do you recover when there is e.g. some connection error and a client misses some events? Can the client ask to replay the events? Then, how far back can they go?

lexicality 5y ago

Feels like a bit silly not to have made this compatible with SSE since all the mechanisms for acessing that are built into browsers these days. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events https://html.spec.whatwg.org/multipage/server-sent-events.ht...

technicolorwhat 5y ago
Yeah, we're also pushing for SSE for this kind of things.
- lexicality 5y ago
  Having a standard for what events look like is a decent idea as people generally just make some random thing up without thinking too hard and having a standard is generally helpful - but making your own polling system etc when there's already a battle tested one available seems like wasted effort and a bit counterproductive
- horstmeyer 5y ago
  Does SSE work reliably in mobile browsers nowadays? A few years back when I tried, it was working OK in Chrome for Android but mobile browser support didn't seem complete.
  - gunapologist99 5y ago
    Sadly, there are very few mobile browsers that aren't Chrome or Safari. In fact, Safari/Webkit is the only mobile browser engine available on IOS at all -- all other browsers are just chrome (pun intended) around IOS webkit. To your point, https://caniuse.com/eventsource https://caniuse.com/eventsource Also there are polyfills for very old browsers like IE.
- mromanuk 5y ago
  I had previously used SSE in a project and had no idea it existed or that it was widely supported by most browsers. It's a beautiful and straightforward technology, but it doesn't seem to be as widely used as it should be, and more tooling is required. Also more tooling is required.

outsomnia 5y ago

How about "catching up" on events, if a client had an outage, can it indicate the last event ID it had and then get a natural replay of the older events?

tirpen 5y ago
That's the second example on the page.
- outsomnia 5y ago
  Doh... thanks for pointing it out. I asked because it says elsewhere that persistence of objects is a non-goal, I guess it meant storing them clientside or something.

deleted 5y ago

[deleted]

chaz6 5y ago

My first thought is if the client does not specify a start ID it could be sent billions of records depending on the elapsed time and frequency of events. What would be the best way to avoid overloading a client?

quaintdev 5y ago
I think the server will give a truncated response. They need to clearly document what is the behaviour for this case.

brasetvik 5y ago

It'd be great to get a fairly standard way of doing this. :) Having worked in this problem space a bit recently, I find this part a bit too optimistic: > The event.id is used as lastEventId to scroll through further events. This means that events need to be strongly ordered to retrieve subsequent events. The example relies on time-ordered UUIDv6 and mentions time sync as a gotcha. This should work well if you only have a single writer. Even with perfectly synced clocks, anything that lets you do _concurrent_ writes can still commit out of order, though. Consider two transactions in a single-node-and-trivially-clock-synced Postgres, for example. If the first transaction that gets the lower timestamp commits after a second transaction that gets a higher timestamp, the second and higher timestamp might've been retrieved by a consumer already (it committed, so it's visible after all), and now you've missed writes. This is also (at least for Postgres, but I guess also in general) true for sequences. The approach I'm currently pursuing involves having an opaque cursor that encodes enough of the MVCC information (i.e. Postgres' txid_current and xip_list) to be able to catch those situations. For a client, the cursor is opaque and they can't see the internals. For the server side, it's quite implementation specific, however. It still has the nice property that clients keep track on where they are, without the server keeping track of where the clients are, which is desirable if the downstream client can roll back e.g. due to recovery/restore from backup) A base64-encoded (possibly encrypted) cursor can wrap whatever implementation specifics are needed and hide them from the client. That implementation could of course be a simple event id if the writing side is strictly serial.

0des 5y ago
Perhaps the time problem can be handled with an eventually consistent Lamport clock system
- jpgvm 5y ago
  Logical/vector clocks etc can be a real pain with ephemeral clients like web browsers, it can be hard to work out when it's safe to trim the now dormant nodes. Though to address the GP a bit, the problem of concurrent writers without a sequencer (like a database) is less common than you might think. It definitely still comes up and there are things like CRDTs to help you address these cases (which do generally rely on either logical clocks or hybrid logical clocks). However most cases of event streaming to the browser you have each write round-tripping through the DB anyway and you use a feed like this to push a CDC stream or similar down into the browser to get "instant" feedback of a change that occurred after initial load.

josephg 5y ago

Very cool! This looks very similar to what we're doing with the braid spec[1], though I really like how clear and concise your examples are! This is a great little website. Some differences between our approaches: - I think its a good idea to support arbitrary patch formats via a content-type style field, just like we have different formats for images. This lets you use the same protocol for things like collaborative editors. - For some data sets (like CRDTs), you want each change to be able to refer to multiple "parents". - Your protocol is very JSON-y and not very HTTP-y. It looks like you're basically using JSON to express HTTP. Why not just use HTTP? One big downside of the JSON approach is that it makes it awkward to transmit binary patches. (Eg, 'patching' an image) Feel free to reach out if you're up for a chat! Looks like we're working on the same problem. [1] https://github.com/braid-org/braid-spec/blob/master/draft-toomim-httpbis-braid-http-03.txt https://github.com/braid-org/braid-spec/blob/master/draft-to...

0des 5y ago
This made me all warm and fuzzy inside that nerds can still help other nerds even when we are on different teams. Hope y'all are able to share some ideas.

rendall 5y ago

This could form the foundation of a distributed social media network.

catwell 5y ago

I wrote a blog post advocating for something like this recently, with an additional push to notify the client when it should poll: https://blog.separateconcerns.com/2022-03-05-push-to-poll.html https://blog.separateconcerns.com/2022-03-05-push-to-poll.ht... I wasn't aware of this but it fits the use case perfectly, I will update the post.

jkarneges 5y ago
Indeed, these kinds of abbreviated notifications are a great way to implement realtime updates. The code is simpler and there's less chance of screwing something up. I've also seen this described as hints: https://blog.andyet.com/2014/09/23/send-hints-not-data/ https://blog.andyet.com/2014/09/23/send-hints-not-data/

wrren 5y ago

I like this idea, however I think the event ID being encoded in the response body places constraints on what each element looks like. Perhaps it would make more sense to encode the last ID/feed position in a response header and have the client submit that in a subsequent request header? That would decouple the feed position from any one element or the response structure itself.

sdze 5y ago

I always implement my own standards. I am faster that way instead of learning what others created. Obviously this is only true for less complex subjects.

adenozine 5y ago
See also: technical debt
- fuckyouwedodata 5y ago
  Not if you keep using your standards across different projects

fuckyouwedodata 5y ago

Is it sensible to keep the TCP connection alive for long periods of time, reading in events as ljson? I imagine that this is no issue for QUIC

jon_adler 5y ago
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- fuckyouwedodata 5y ago
Karrot_Kream 5y ago
Yeah there's no problems with a long-lived TCP connection unless one end of the connection tries to drop the connection after a certain amount of time (which certain firewalls do.)
- fuckyouwedodata 5y ago
  Sounds good. Thanks for pointing out the issue with some firewalls

pfsalter 5y ago

This seems similar, but not as fleshed out as RPDE feeds, which has already been accepted by the W3 [1], and provides many answers to lots of the 'eventually up to date' questions. [1] https://www.w3.org/2017/08/realtime-paged-data-exchange/ https://www.w3.org/2017/08/realtime-paged-data-exchange/

rmetzler 5y ago

No pagination?

Fiahil 5y ago
You probably mean "maximum batch size", because pagination is handled by the lastEventId and the fact that a stream is always consumed in order.
aiobe 5y ago
Correct. No fixed sized pages, but dynamic batches based on the lastEventId. This is much easier to implement, both in server and client side, and it greatly removed the amount of data transferred. With fixed pages you would return content of the latest page for every poll request until it is "full".

xg15 5y ago

It's an interesting idea, but I wonder how well the promise will hold up that this is less complex than using a message broker. Yes, the network topology and protocol are certainly less complex, but there are now additional strong requirements how an endpoint has to store and manage existing events. (See the bits about strict time ordering, compaction, aggregation, etc). A lot of this is effectively what a message broker is doing in a "traditional" system to guarantee consistency. Those tasks aren't gone, they are just pushed to the endpoints now.

efunneko 5y ago
My thoughts exactly. While this is more human readable on the wire, a message broker delivering the feed would provide many different other features that might be useful, such as transactions, load-balancing, guaranteed delivery and per-endpoint state to simplify the individual application instances. For those not aware of what message brokers are, there are many to choose from such as: Mosquito, RabbitMQ, ActiveMQ, Solace... If delivery over HTTP is a requirement, many of these brokers support delivery over websockets or (in the case of Solace) also support long polling.
aiobe 5y ago
A few issues with message brokers, esp. in the system-to-system integration: - Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... - Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, ... Who is on call when something goes wrong? - Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting...
- xg15 5y ago
  > Security: In B2B scenarios or public APIs would you open your broker to the WWW? HTTP has a solid infrastructure, including firewalls, ddos defence, API gateways, certificate management, ... That's a valid point. I think it's a pity we don't have an equivalent standard for asynchronous messaging with the same support as HTTP. However, there are lots of options for presenting an asynchronous public API that would use your message broker behind the scenes, without fully exposing it: Websockets, SSE, web hooks, etc... > Organisational dependencies: Some team needs to maintain the broker (team 1, team 2, or a third platform team). You have a dependency to this team, if you need a new topic, user, True, but don't you have that anyway? How is this different from requesting a new database, service route, service definition, etc? > Technology ingestion: A message broker ingests technology into the system. You need compatible client libraries, handle version upgrades, resilience concepts, learn troubleshooting... How simple or complex this is depends on the concrete broker at hand. There are some protocols, e.g. STOMP that are simple enough that you could write your own client. And as I wrote in the parent: HTTP feeds are a technology as well. You'll have to think about troubleshooting and resiliency there as well.

xg15 5y ago

With more attention going to long polling again, I wonder if it would be useful to introduce some kind of HTTP signaling (header or 1xx status) to indicate on the protocol level that long polling is going on. This might be useful information for intermediaries - e.g. proxies, firewalls, browser network tab, etc.

yashasolutions 5y ago

This looks great. Has anyone any insights or reference about the limits of this approach compared to more traditional message queues?

dgritsko 5y ago
One obvious downside is that polling at a fixed interval is going to be less efficient than having an event "pushed" to your client as soon as it's available. If your events are low-volume, then there will be periods where your polling requests return no new data, but you still have to make those requests anyway in order to determine that. And conversely, if your events are high-volume, then your polling interval represents an arbitrary amount of delay that you're introducing into the system. That might not be big deal for some applications, but it's worth mentioning as a potential downside in situations where you want the behavior to be as near-instantaneous as possible.

diordiderot 5y ago

Can someone ELI5 the significance of this? * What are some scenarios where you would need a feed? * What's being done now to solve the problem and how is this different? Many thanks

kitd 5y ago
This basically puts a REST/HTTP GET frontend on your messaging backend (Kafka, RabbitMQ, MQTT, etc). So whatever events you want to expose publicly can be done over plain HTTP. Having said that, IMO using SSE or websockets would be a better fit than raw HTTP.
tiernano 5y ago
I could see this being used as part of a distributed system for shopping. for example, say you have 20 stores, each with its own storage area for items, but you also have 2 large warehouses. Your online site might need to know which stores have stock now, or which warehouses have stock. The site might allow you to do "Click and collect" do it would need to know, in somewhat real time, which stores actually have the stock. each store would have its own endpoint with a data feed that the central server can get data from. Same with online orders for delivery. It needs to know what warehouses have the stock, and if none do, how to get the store to ship it to a customer. Likewise, the stores might need to know what is in the warehouses, or even across town; someone walks in store to order something, but its not in that store. But they know, in real time, that its in the warehouse for delivery next day or that the store across the city has it. What is done now is a more centralised approach; all sites would have a connection to a single DB in head office that stores everything. This makes things more distributed and, in theory, removes a single point of failure. I should clarify, I am not working on any of this, this is just how I think it would work... if anyone wants to step in and tell me if I'm wrong, right, or just plan stupid, please shout.
- diordiderot 5y ago
  Great answer and exactly what I was looking for! Thank you

yencabulator 5y ago

I think serving data from multiple Kafka partitions gets unnecessarily hard if your "continue from this point" token is tied to a singular event ID. For that, it'd be better to have the "cursor token" be an arbitrary blob of data you repeat back to the server. Then the server can e.g. encode a list of partition->offset values into it.