6 ms·

HTTP Feeds: a minimal specification for polling events over HTTP

72 points by sea-gold 1y ago

sea-gold 1y ago
Previously discussed (April 2022; 95 comments): https://news.ycombinator.com/item?id=30904220 https://news.ycombinator.com/item?id=30904220
hombre_fatal 1y ago
It could use a section on high level justification / inspiration. For example, what inspired this over a typical paginated API that lets you sort old to new with an afterId parameter?
lud_lite 1y ago
What happens if you need to catch up? You keep calling in a loop with a new lastEventId? What is the intention there though. Is this for social media type feeds, or is this meant for synchronising data (at the extreme for DB replication for example!). What if anything is expected of the producer in terms of how long to store events?
- dgoldstein0 1y ago
  Sounds like it. But the compaction section has more details - basically you can discard events that are overwritten by later ones
fefe23 1y ago
This is an astonishingly bad idea. Don't do this. Use HTTP server-sent events instead. Those can keep the connection open so you don't have to poll to get real-time updates and they will also let you resume from the last entry you saw previously. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...
- montroser 1y ago
  Yeah, but in real life, SSE error events are not robust, so you still have to do manual heartbeat messages and tear down and reestablish the connection when the user changes networks, etc. In the end, long-polling with batched events is not actually all that different from SSE with ping/pong heartbeats, and with long-polling, you get the benefit of normal load balancing and other standard HTTP things
  - xyzzy_plugh 1y ago
    Correct. In the end, mechanically, nothing beats long polling. Everything ends up converging at which point you may as well just long poll.
  - mikojan 1y ago
    But SSE is a standard HTTP thing. Why would you not be able to do "normal load balancing"? I would also rather not have a handful of long-polling loops pollute the network tab.
    - jpc0 1y ago
      “Normal load balancing” means “Request A goes to server A”, “Request B goes to server B” and there is no state held in the server, if there is a session its stored in a KV store or database which persists. With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Some proxies / load balancers don’t like long lasting connections and will tear them down if there has been no traffic so your need to constantly send a heart beat. I have deployed SSE, I love the technology, I wouldn’t deploy it if I don’t control the end devices and everything in between, I would just do long polling.
      - kiitos 1y ago
        Your description of "normal load balancing" is certainly one way to do load balancing, but in no way is it the presumptive default. Keeping session data in a shared source of truth like a KV store or DB, and expecting (stateless) application servers to do all their session stuff thru that single source of truth, is a fine approach for some use cases, but certainly not a general-purpose solution. > With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Weird take. SSE is inherently stateful, sure, in the sense that it generally expects there to be a single long-lived connection between the client and the server, thru which events are emitted. Purpose of that being that it's a more efficient way to stream data from server to client -- for specific use cases -- than having the client long-poll on an endpoint.
wackget 1y ago
Did someone just reinvent a GET API with cursor-based pagination?
- hdjrudni 1y ago
  Sure looks like it. I'm not getting what's new or interesting here. Cursors are actually better because you can put any kind of sort order in there. This "lastEventId" seems to be strictly chronological.
DidYaWipe 1y ago
Never heard of "CloudEvents" before. How do people feel about those?
- paulddraper 1y ago
  Good
- mdaniel 1y ago
  I enjoy anything that drives down NIH, or something that an existing library could possibly support, or something that I could take to my next job (or could possibly hire for) I believe cloud events are most common in Kafka-adjacent or event-driven architectures but I think they're used in some GCP serverless things, too
  - Maxious 1y ago
    They're also documented for aws https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations-cloudevents.html https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-... and argo-events https://github.com/argoproj/argo-events/blob/master/docs/concepts/event_source.md https://github.com/argoproj/argo-events/blob/master/docs/con...
  - DidYaWipe 1y ago
    I guess you mean "not invented here..." Yep, any opportunity to not reinvent stuff is a big win. But I'm wary of layers upon layers. I'm thinking of how this could be combined with MQTT... it doesn't seem totally redundant.
- junto 1y ago
  Here’s a nice comparison between CloudEvents and AsyncAPI from 2019. You can combine them. In the end being able to version and wrap events is useful, although amusingly it reminds me of SOAP. https://www.asyncapi.com/blog/asyncapi-cloud-events https://www.asyncapi.com/blog/asyncapi-cloud-events
  - DidYaWipe 1y ago
philsnow 1y ago
Because the client requests pagination by lastEventId (a UUID), the server needs to remember every event forever in order to correctly catch up clients. If instead the client paginated by lastEventTimestamp, then a server that for any reason no longer had a particular event UUID could at least start at the following one.
- tinodb 1y ago
  That’s why the article suggests using a uuid v6 which is time orderable. Or prefixing with an incrementing db id. So indeed, if you intend to delete events, you might want to make sure you have orderable ids of some sort.
zzo38computer 1y ago
I think that HTTP is not the best way to do it, and that JSON is also not the best way to do it. (HTTP may work reasonably when you only want to download existing events and do not intend to continue polling.) I also think using UUID alone isn't the best way to make the ID number. If events only come from one source, then just using autoincrementing will work (like NNTP does for article numbers within a group); being able to request by time might also work (which is also something that NNTP does).

72 points by sea-gold 1y ago

Previously discussed (April 2022; 95 comments): https://news.ycombinator.com/item?id=30904220 https://news.ycombinator.com/item?id=30904220

hombre_fatal 1y ago

It could use a section on high level justification / inspiration. For example, what inspired this over a typical paginated API that lets you sort old to new with an afterId parameter?

lud_lite 1y ago

What happens if you need to catch up? You keep calling in a loop with a new lastEventId? What is the intention there though. Is this for social media type feeds, or is this meant for synchronising data (at the extreme for DB replication for example!). What if anything is expected of the producer in terms of how long to store events?

dgoldstein0 1y ago
Sounds like it. But the compaction section has more details - basically you can discard events that are overwritten by later ones

fefe23 1y ago

This is an astonishingly bad idea. Don't do this. Use HTTP server-sent events instead. Those can keep the connection open so you don't have to poll to get real-time updates and they will also let you resume from the last entry you saw previously. https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...

montroser 1y ago
Yeah, but in real life, SSE error events are not robust, so you still have to do manual heartbeat messages and tear down and reestablish the connection when the user changes networks, etc. In the end, long-polling with batched events is not actually all that different from SSE with ping/pong heartbeats, and with long-polling, you get the benefit of normal load balancing and other standard HTTP things
- xyzzy_plugh 1y ago
  Correct. In the end, mechanically, nothing beats long polling. Everything ends up converging at which point you may as well just long poll.
- mikojan 1y ago
  But SSE is a standard HTTP thing. Why would you not be able to do "normal load balancing"? I would also rather not have a handful of long-polling loops pollute the network tab.
  - jpc0 1y ago
    “Normal load balancing” means “Request A goes to server A”, “Request B goes to server B” and there is no state held in the server, if there is a session its stored in a KV store or database which persists. With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Some proxies / load balancers don’t like long lasting connections and will tear them down if there has been no traffic so your need to constantly send a heart beat. I have deployed SSE, I love the technology, I wouldn’t deploy it if I don’t control the end devices and everything in between, I would just do long polling.
    - kiitos 1y ago
      Your description of "normal load balancing" is certainly one way to do load balancing, but in no way is it the presumptive default. Keeping session data in a shared source of truth like a KV store or DB, and expecting (stateless) application servers to do all their session stuff thru that single source of truth, is a fine approach for some use cases, but certainly not a general-purpose solution. > With SSE the server has to be stateful, for load balancing to work you need to be able to migrate connections between servers. Weird take. SSE is inherently stateful, sure, in the sense that it generally expects there to be a single long-lived connection between the client and the server, thru which events are emitted. Purpose of that being that it's a more efficient way to stream data from server to client -- for specific use cases -- than having the client long-poll on an endpoint.

wackget 1y ago

Did someone just reinvent a GET API with cursor-based pagination?

hdjrudni 1y ago
Sure looks like it. I'm not getting what's new or interesting here. Cursors are actually better because you can put any kind of sort order in there. This "lastEventId" seems to be strictly chronological.

DidYaWipe 1y ago

Never heard of "CloudEvents" before. How do people feel about those?

paulddraper 1y ago
Good
mdaniel 1y ago
I enjoy anything that drives down NIH, or something that an existing library could possibly support, or something that I could take to my next job (or could possibly hire for) I believe cloud events are most common in Kafka-adjacent or event-driven architectures but I think they're used in some GCP serverless things, too
- Maxious 1y ago
  They're also documented for aws https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations-cloudevents.html https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-... and argo-events https://github.com/argoproj/argo-events/blob/master/docs/concepts/event_source.md https://github.com/argoproj/argo-events/blob/master/docs/con...
- DidYaWipe 1y ago
  I guess you mean "not invented here..." Yep, any opportunity to not reinvent stuff is a big win. But I'm wary of layers upon layers. I'm thinking of how this could be combined with MQTT... it doesn't seem totally redundant.
junto 1y ago
Here’s a nice comparison between CloudEvents and AsyncAPI from 2019. You can combine them. In the end being able to version and wrap events is useful, although amusingly it reminds me of SOAP. https://www.asyncapi.com/blog/asyncapi-cloud-events https://www.asyncapi.com/blog/asyncapi-cloud-events
- DidYaWipe 1y ago

philsnow 1y ago

Because the client requests pagination by lastEventId (a UUID), the server needs to remember every event forever in order to correctly catch up clients. If instead the client paginated by lastEventTimestamp, then a server that for any reason no longer had a particular event UUID could at least start at the following one.

tinodb 1y ago
That’s why the article suggests using a uuid v6 which is time orderable. Or prefixing with an incrementing db id. So indeed, if you intend to delete events, you might want to make sure you have orderable ids of some sort.

zzo38computer 1y ago

I think that HTTP is not the best way to do it, and that JSON is also not the best way to do it. (HTTP may work reasonably when you only want to download existing events and do not intend to continue polling.) I also think using UUID alone isn't the best way to make the ID number. If events only come from one source, then just using autoincrementing will work (like NNTP does for article numbers within a group); being able to request by time might also work (which is also something that NNTP does).