3 ms·

Web hooks are not reliable. When engineering your system, you need to implement a scheduled based watcher to go back and clean up hook issues.

bob1029 3y ago
I gave up trying to be a polite API consumer with GitHub events, etc. Polling the basic resources every minute is way more reliable and your code will survive a junior developer's shenanigans. I had the REST API e-tag polling working well, but then I discovered my org event stream didn't include label changes. This is a separate thing I needed to poll and at that point I lost my mind. I refuse to keep track of 6+ pieces of state in order to pull essential data from an API. My current pattern is to list all open issues and then compare their updated_at with persisted copies. If any changes, then I refresh the comments for the issue as well as top-level items (title/body/labels).
- calvinmorrison 3y ago
  It's part of the Shopify and BigCommerce's design pattern explicity. Like, yes we offer webhooks, but also you need to come back once an X (I typically do hourly) and sweep for missed data. Not to mention with Shopify, webhooks are not guaranteed to be at all ordered. You may receive an order.updated event prior to an order.create event. Delivery milage and timing may vary. So, after all that back and forth we asked ourselves, why do we even bother with webhooks? Well for one, it helps our clients being real time to improve their shipping speed, keeping real time inventory, and so forth. Secondly, it keeps our processing more flat. We might run an hourly cron to pickup stragglers, but we don't do huge data dumps per hour. In some our busier systems it can be a real challenge when clients do things like monthly invoicing all their customers. Using webhooks and processing records as they come in keeps the queue as small as reasonable.
brazzledazzle 3y ago
Yeah GitHub doesn't like when you do frequent polling over hooks but you pretty much need to have something polling (infrequently) to catch undelivered hooks.

bob1029 3y ago

I gave up trying to be a polite API consumer with GitHub events, etc. Polling the basic resources every minute is way more reliable and your code will survive a junior developer's shenanigans. I had the REST API e-tag polling working well, but then I discovered my org event stream didn't include label changes. This is a separate thing I needed to poll and at that point I lost my mind. I refuse to keep track of 6+ pieces of state in order to pull essential data from an API. My current pattern is to list all open issues and then compare their updated_at with persisted copies. If any changes, then I refresh the comments for the issue as well as top-level items (title/body/labels).

calvinmorrison 3y ago
It's part of the Shopify and BigCommerce's design pattern explicity. Like, yes we offer webhooks, but also you need to come back once an X (I typically do hourly) and sweep for missed data. Not to mention with Shopify, webhooks are not guaranteed to be at all ordered. You may receive an order.updated event prior to an order.create event. Delivery milage and timing may vary. So, after all that back and forth we asked ourselves, why do we even bother with webhooks? Well for one, it helps our clients being real time to improve their shipping speed, keeping real time inventory, and so forth. Secondly, it keeps our processing more flat. We might run an hourly cron to pickup stragglers, but we don't do huge data dumps per hour. In some our busier systems it can be a real challenge when clients do things like monthly invoicing all their customers. Using webhooks and processing records as they come in keeps the queue as small as reasonable.

brazzledazzle 3y ago

Yeah GitHub doesn't like when you do frequent polling over hooks but you pretty much need to have something polling (infrequently) to catch undelivered hooks.