4 ms·

Yeah, one of our goals was to not add a new/weaker point of failure. We gracefully fall back to S3 directly if our cache server is down without a hiccup. So th

Yeah, one of our goals was to not add a new/weaker point of failure. We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a failure, we'd go back to slightly degraded performance by talking across the country until we brought it back online.

wimagguc 10y ago
Maybe I missed this in the docs, but why isn't HAProxy considered a new point of failure?
- mattrobenolt 10y ago
  It's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a network issue between machines, etc.
  - wimagguc 10y ago
    Sure, that's well understood. Being a low-risk point of failure, isn't it still a new one? It does come with setup and maintenance costs, test scenarios etc., so it's only fair to recognise this as a risk.
    - matt_wulfeck 10y ago
      Don't worry, they run HAproxy in front of haproxy incase the haproxy to S3 service goes down.
    - mattrobenolt 10y ago
      Technically yes. But we're pretty accustomed to this level of risk. For something like this, the pros far outweigh the cons involved. Yeah, it could fail. The maintenance overhead of this is absolutely minimal and took a handful of hours to have tested and in production. Also worth noting, that this isn't really a single point of failure as a system wide thing. It'd only be a single point of failure on that single node. So if haproxy decided to explode, only that one machine would have a problem momentarily, while the process got started back up with our process manager. The worst case scenario is a human error where we ship a bad config and break everything.
      - greenleafjacob 10y ago
        Not really true. If you for example mistune maxconn haproxy will stop accepting new connections and that's likely to happen cluster wide.
        mattrobenolt 10y ago
        This is equivalent to shipping bad application code that takes everything down. Except the config is only a handful of lines of code and will very likely never change again. Also, we don't blindly roll out changes cluster wide for things like this without testing explicitly on staging or test nodes.
        datums 10y ago
        Shipping it with the app, you lose the cluster wide cached objects. A SPOF is the resolver. It's google but it's a SPOF. Is the failover to s3 automatic ? Or do you make a code change ? What kind of latency does that add ?
        nalllar 10y ago
        Haproxy is localhost. Caching nginx is nearby but not local, so the cache is shared. Haproxy sends to caching nginx if available, else directly to s3.
        datums 10y ago
        Thx. Got it. I don't know enough about the app, but I would have it serve directly from CF to users. Instead of hitting this environment for static assets. Good job and good conversation.
- justinsaccount 10y ago
  > Each application server that’s running our Sentry code has an HAProxy process running on localhost.
newobj 10y ago
The idea that it being a local proxy meaning there is no operational overhead is a dangerous fallacy. If that's an earnestly literal statement from you, then it means you simply haven't encountered the failure modes that these kinds of set up are inclined towards. I've worked at several BigCo's, seen them all implement this pattern, and seen every single one of them have fleet-wide outages due to these innocent "local proxies". Remember FB's 2-3 hour outage 2 years ago? https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/ https://www.facebook.com/notes/facebook-engineering/more-det... It was /exactly/ this kind of "local proxy for higher availability/caching over the downstream thing" that caused the outage.
- newobj 10y ago
  (To be clear, I'm not saying it's an anti-pattern, I'm just saying that calling it "no operational overhead" is naive)
  - wmccullough 10y ago
    Mince words all you want, they stayed operational when many others did not. That's success in my book.
    - soft_dev_person 10y ago
      For an increased risk of going down when everybody else is up. Is that still success? And at what other costs? It all comes down to risk vs. cost vs. gain.
- mattrobenolt 10y ago
  Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failure. If this has a problem, we'll also have problems talking to other services in our network. And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.
arrty88 10y ago
You are assuming your haproxy server is always operational, no?
- mattrobenolt 10y ago
  It's running on localhost, so it's not it's own machine. It's local to the servers running the application code.

wimagguc 10y ago

Maybe I missed this in the docs, but why isn't HAProxy considered a new point of failure?

mattrobenolt 10y ago
It's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a network issue between machines, etc.
- wimagguc 10y ago
  Sure, that's well understood. Being a low-risk point of failure, isn't it still a new one? It does come with setup and maintenance costs, test scenarios etc., so it's only fair to recognise this as a risk.
  - matt_wulfeck 10y ago
    Don't worry, they run HAproxy in front of haproxy incase the haproxy to S3 service goes down.
  - mattrobenolt 10y ago
    Technically yes. But we're pretty accustomed to this level of risk. For something like this, the pros far outweigh the cons involved. Yeah, it could fail. The maintenance overhead of this is absolutely minimal and took a handful of hours to have tested and in production. Also worth noting, that this isn't really a single point of failure as a system wide thing. It'd only be a single point of failure on that single node. So if haproxy decided to explode, only that one machine would have a problem momentarily, while the process got started back up with our process manager. The worst case scenario is a human error where we ship a bad config and break everything.
    - greenleafjacob 10y ago
      Not really true. If you for example mistune maxconn haproxy will stop accepting new connections and that's likely to happen cluster wide.
      - mattrobenolt 10y ago
        This is equivalent to shipping bad application code that takes everything down. Except the config is only a handful of lines of code and will very likely never change again. Also, we don't blindly roll out changes cluster wide for things like this without testing explicitly on staging or test nodes.
        datums 10y ago
        Shipping it with the app, you lose the cluster wide cached objects. A SPOF is the resolver. It's google but it's a SPOF. Is the failover to s3 automatic ? Or do you make a code change ? What kind of latency does that add ?
        nalllar 10y ago
        Haproxy is localhost. Caching nginx is nearby but not local, so the cache is shared. Haproxy sends to caching nginx if available, else directly to s3.
        datums 10y ago
        Thx. Got it. I don't know enough about the app, but I would have it serve directly from CF to users. Instead of hitting this environment for static assets. Good job and good conversation.
justinsaccount 10y ago
> Each application server that’s running our Sentry code has an HAProxy process running on localhost.

newobj 10y ago

The idea that it being a local proxy meaning there is no operational overhead is a dangerous fallacy. If that's an earnestly literal statement from you, then it means you simply haven't encountered the failure modes that these kinds of set up are inclined towards. I've worked at several BigCo's, seen them all implement this pattern, and seen every single one of them have fleet-wide outages due to these innocent "local proxies". Remember FB's 2-3 hour outage 2 years ago? https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/ https://www.facebook.com/notes/facebook-engineering/more-det... It was /exactly/ this kind of "local proxy for higher availability/caching over the downstream thing" that caused the outage.

newobj 10y ago
(To be clear, I'm not saying it's an anti-pattern, I'm just saying that calling it "no operational overhead" is naive)
- wmccullough 10y ago
  Mince words all you want, they stayed operational when many others did not. That's success in my book.
  - soft_dev_person 10y ago
    For an increased risk of going down when everybody else is up. Is that still success? And at what other costs? It all comes down to risk vs. cost vs. gain.
mattrobenolt 10y ago
Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failure. If this has a problem, we'll also have problems talking to other services in our network. And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.

arrty88 10y ago

You are assuming your haproxy server is always operational, no?

mattrobenolt 10y ago
It's running on localhost, so it's not it's own machine. It's local to the servers running the application code.