4 ms·

Building polling infrastructure is substantially more complex than setting up a GET endpoint to handle webhooks. If you need strict consistency guarantees, sur

Building polling infrastructure is substantially more complex than setting up a GET endpoint to handle webhooks. If you need strict consistency guarantees, sure. Otherwise, don't piss off your API consumers, webhooks work just fine.

crooked-v 5y ago
> Building polling infrastructure is substantially more complex That really depends on the setup you're using. There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler.
- sergiomattei 5y ago
  I'm looking at it from the perspective of a web dev. If you're already building a web application, it is almost always more work to do the polling setup. In Python, it's either a cron script or Celery beat. In PHP, usually a cron script. That all means more processes running outside of your web framework, more stuff to deal with, more complexity. Now you have to manage e.g. running a celery process, managing crontabs wherever you deploy... Even in languages like Elixir where long-running scheduled processes are cheap and native, you still have to write a GenServer in comparison to just using your web framework. I'm fine with the approach in the article. I actually like that they're giving the option for multiple ways of getting those events. Just please, please don't make /events the only option...
  - justsomeuser 5y ago
    I actually have the opposite opinion. I'm an experienced web developer. Setting up polling is easier for me than setting up an HTTP endpoint. I typically use a language with an event loop (async/await), so there is always something like `setInterval(poll, 500)`. All that code needs is a connection to the internet. If the server is down for 24 hours I can start it up and it'll read the missed events. I can set up 10 dev environments with just an API key difference in the config. I can batch apply events in a single database transaction, ensuring consistency. But with a webhook, I need to ensure that my server can accept incoming connections, the external API knows that location. Each dev env needs to replicate this public HTTP server set up. I need to monitor the uptime of the server closely as missing events or erroring on a subset of events could leave my database in an inconsistent state.
- masklinn 5y ago
  > There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler. Or way simpler if you're not building a web application in the first place.
shadowgovt 5y ago
It's less about cost of building and more about cost of running. Polling is chatty when updates are infrequent.
toomuchtodo 5y ago
> If you need strict consistency guarantees, sure. Otherwise, don't piss off your API consumers, webhooks work just fine. Ahh, but all problems are queues. What happens if your webhook destination is down and your source system sending queue backs up? What happens if your destination endpoint is up, providing 200s to requests, but throwing away the data quietly due to a mistake during a rollout? Or your webhook source quickly ramps to a volume that is effectively a DoS attack? (These are all problems I encountered in a role at a low code/no code product). I strongly endorse others in this thread who indicate long polling events is a suitable pattern. You want the best worlds of data durability and consistency through polling functionality, but also as close to real time event firing as possible. Of course, some APIs you have no control over, and are stuck building robust, chatty polling infra to support because your (paying) users demand access to those APIs. Such is the schlep.
- superbaconman 5y ago
  Why do I need two different ways of retrieving events? Wouldn't the data also be available via the usual rest api? If events fail on the consumer end it's their responsibility to resync. Presumably code was written for an initial sync anyway no?

crooked-v 5y ago

> Building polling infrastructure is substantially more complex That really depends on the setup you're using. There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler.

sergiomattei 5y ago
I'm looking at it from the perspective of a web dev. If you're already building a web application, it is almost always more work to do the polling setup. In Python, it's either a cron script or Celery beat. In PHP, usually a cron script. That all means more processes running outside of your web framework, more stuff to deal with, more complexity. Now you have to manage e.g. running a celery process, managing crontabs wherever you deploy... Even in languages like Elixir where long-running scheduled processes are cheap and native, you still have to write a GenServer in comparison to just using your web framework. I'm fine with the approach in the article. I actually like that they're giving the option for multiple ways of getting those events. Just please, please don't make /events the only option...
- justsomeuser 5y ago
  I actually have the opposite opinion. I'm an experienced web developer. Setting up polling is easier for me than setting up an HTTP endpoint. I typically use a language with an event loop (async/await), so there is always something like `setInterval(poll, 500)`. All that code needs is a connection to the internet. If the server is down for 24 hours I can start it up and it'll read the missed events. I can set up 10 dev environments with just an API key difference in the config. I can batch apply events in a single database transaction, ensuring consistency. But with a webhook, I need to ensure that my server can accept incoming connections, the external API knows that location. Each dev env needs to replicate this public HTTP server set up. I need to monitor the uptime of the server closely as missing events or erroring on a subset of events could leave my database in an inconsistent state.
masklinn 5y ago
> There are plenty of server platforms out there where setting up a cron is no more complicated than making a GET route handler. Or way simpler if you're not building a web application in the first place.

shadowgovt 5y ago

It's less about cost of building and more about cost of running. Polling is chatty when updates are infrequent.

toomuchtodo 5y ago

> If you need strict consistency guarantees, sure. Otherwise, don't piss off your API consumers, webhooks work just fine. Ahh, but all problems are queues. What happens if your webhook destination is down and your source system sending queue backs up? What happens if your destination endpoint is up, providing 200s to requests, but throwing away the data quietly due to a mistake during a rollout? Or your webhook source quickly ramps to a volume that is effectively a DoS attack? (These are all problems I encountered in a role at a low code/no code product). I strongly endorse others in this thread who indicate long polling events is a suitable pattern. You want the best worlds of data durability and consistency through polling functionality, but also as close to real time event firing as possible. Of course, some APIs you have no control over, and are stuck building robust, chatty polling infra to support because your (paying) users demand access to those APIs. Such is the schlep.

superbaconman 5y ago
Why do I need two different ways of retrieving events? Wouldn't the data also be available via the usual rest api? If events fail on the consumer end it's their responsibility to resync. Presumably code was written for an initial sync anyway no?