4 ms·

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

by Thaxll 4y ago

HL33tibCe7 4y ago
The latter (they openly admit as such)
- pcthrowaway 4y ago
  They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don't work as advertised. I still think my company is going to take another stab at consul in the future, because we do need service discovery. But they're advertising a solution to an incredibly hard problem with a shit ton of variations in network topology and infra that it should (theoretically) work on. I imagine if you stay on the happy path everything works out just fine with Consul (even then, maybe only most of the time). The problem is that they don't spell out what the happy path is, and that all the other knobs they expose off to the side are actually down paths beleagured by dragons.
  - fastest963 4y ago
    To add a data point we've been using Consul globally for several years now without any major outages. We do close to 50k qps with Consul at peak running on single digit cores per DC.
    - sidlls 4y ago
      50k qps on consul or system wide?
      - fastest963 4y ago
        Consul specifically
  - jen20 4y ago
    Out of interest why would one presume that they are being actively supported? I haven’t read everything about this saga, but I’ve never seen any mention of a commercial relationship.
    - pcthrowaway 4y ago
      If you're on a slowly-sinking ship, it'd be silly not to at least try to bring in someone with the full context of the ship's architecture to get it serviceable before going all-in on the decision to engineer and build an entirely new kind of ship while still aboard the sinking one.
      - jen20 4y ago
        That’s not really what I asked: do you have any actual evidence of a commercial relationship rather than the notion that it wouldn’t be a bad idea?
abofh 4y ago
The former ish -- they relied on consul marketing that the hammer fit the square hole. Hashicorp has been pretty bad about marketing themselves as the right tool for any job, but they really only fit the narrowest of tasks before you find yourself needing an alternative or being compelled to buy a support contact. It's atlassian from Arkansas, just faster
- akerl_ 4y ago
  Can you cite this? Because it seems like the opposite is true: https://news.ycombinator.com/item?id=35048318 https://news.ycombinator.com/item?id=35048318
  - abofh 4y ago
    You seem to have cited exactly why I don't recommend my clients use a hashi stack, it seems like you've failed to make a point?
    - akerl_ 4y ago
      > I have only positive things to say about every HashiCorp product I've worked with since I got here. You're making a claim that Hashicorp sold themselves as being a solution for problems that they can't solve. But the comment from an actual Fly.io employee suggests that isn't the case. They're stating that Fly.io pushed the product beyond its limits, and they don't seem to be projecting any of that as being the fault of Hashicorp the company or of their products.
      - abofh 4y ago
        You literally started with a quote that I didn't write. Hashi is dishonest and diminuative, providing products that generally should've been written off as a loss Edit: We're apparently not allowed to interact beyond three replies: I have no beef with any hashi product that is satisfactory. Your reply quotation explanation is unsatisfactory, you tried to quote something into a thread in the most irresponsible way you could - got called out for it and tried to top post to make it work. I have never had a client using a hashi stack that was happy about it: price, quality or reliability it's a failure I don't begrudge their work, their work is just subpar Quote that if you'd like, I won't interact with someone that starts with fraudulent misrepresentation.
sidlls 4y ago
Both, though the latter is likely due to marketing/promises from HashiCorp. Consul (and the entire hashicorp stack, really) is overengineered, under-optimized, and generally terrible to use at any scale beyond "small".
- jen20 4y ago
  And yet the comment from a member of the actual team in question underneath says the opposite…
berkle4455 4y ago
Roblox had a massive 3-day outage [1] in October 2021 due to a Consul feature that didn’t work as expected. My gut with Consul is don’t use it for high-load distributed services. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
- jimmyl02 4y ago
  The Roblox outage seemed like a pretty one-off instance due to a hard to catch bug. Consul still seems like a great choice and it looks like Roblox continues to use it at their scale.
mrkurt 4y ago
The design of Consul is wrong for what we need to do. Consul has been pretty good when it's running, but it's a huge pain in the ass to recover when it falls over. And when it does fall over, it's usually with no notice.
seabrookmx 4y ago
Why not both?

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

by Thaxll 4y ago

They seem to have a lot of issues with Consul, is it the design of Consul or the way they use it that is the problem?

HL33tibCe7 4y ago

The latter (they openly admit as such)

pcthrowaway 4y ago
They say that, but they're also being actively supported by Hashicorp right now (one would presume), so they really need to maintain a good working relationship. I don't have a relationship with Hashicorp, and have tried using Consul. Everything about it is amazing in theory, but you might need a few years of experience with kube, consul, go, and maybe even the hashicorp stack to even begin debugging when things don't work as advertised. I still think my company is going to take another stab at consul in the future, because we do need service discovery. But they're advertising a solution to an incredibly hard problem with a shit ton of variations in network topology and infra that it should (theoretically) work on. I imagine if you stay on the happy path everything works out just fine with Consul (even then, maybe only most of the time). The problem is that they don't spell out what the happy path is, and that all the other knobs they expose off to the side are actually down paths beleagured by dragons.
- fastest963 4y ago
  To add a data point we've been using Consul globally for several years now without any major outages. We do close to 50k qps with Consul at peak running on single digit cores per DC.
  - sidlls 4y ago
    50k qps on consul or system wide?
    - fastest963 4y ago
      Consul specifically
- jen20 4y ago
  Out of interest why would one presume that they are being actively supported? I haven’t read everything about this saga, but I’ve never seen any mention of a commercial relationship.
  - pcthrowaway 4y ago
    If you're on a slowly-sinking ship, it'd be silly not to at least try to bring in someone with the full context of the ship's architecture to get it serviceable before going all-in on the decision to engineer and build an entirely new kind of ship while still aboard the sinking one.
    - jen20 4y ago
      That’s not really what I asked: do you have any actual evidence of a commercial relationship rather than the notion that it wouldn’t be a bad idea?

abofh 4y ago

The former ish -- they relied on consul marketing that the hammer fit the square hole. Hashicorp has been pretty bad about marketing themselves as the right tool for any job, but they really only fit the narrowest of tasks before you find yourself needing an alternative or being compelled to buy a support contact. It's atlassian from Arkansas, just faster

akerl_ 4y ago
Can you cite this? Because it seems like the opposite is true: https://news.ycombinator.com/item?id=35048318 https://news.ycombinator.com/item?id=35048318
- abofh 4y ago
  You seem to have cited exactly why I don't recommend my clients use a hashi stack, it seems like you've failed to make a point?
  - akerl_ 4y ago
    > I have only positive things to say about every HashiCorp product I've worked with since I got here. You're making a claim that Hashicorp sold themselves as being a solution for problems that they can't solve. But the comment from an actual Fly.io employee suggests that isn't the case. They're stating that Fly.io pushed the product beyond its limits, and they don't seem to be projecting any of that as being the fault of Hashicorp the company or of their products.
    - abofh 4y ago
      You literally started with a quote that I didn't write. Hashi is dishonest and diminuative, providing products that generally should've been written off as a loss Edit: We're apparently not allowed to interact beyond three replies: I have no beef with any hashi product that is satisfactory. Your reply quotation explanation is unsatisfactory, you tried to quote something into a thread in the most irresponsible way you could - got called out for it and tried to top post to make it work. I have never had a client using a hashi stack that was happy about it: price, quality or reliability it's a failure I don't begrudge their work, their work is just subpar Quote that if you'd like, I won't interact with someone that starts with fraudulent misrepresentation.

sidlls 4y ago

Both, though the latter is likely due to marketing/promises from HashiCorp. Consul (and the entire hashicorp stack, really) is overengineered, under-optimized, and generally terrible to use at any scale beyond "small".

jen20 4y ago
And yet the comment from a member of the actual team in question underneath says the opposite…

berkle4455 4y ago

Roblox had a massive 3-day outage [1] in October 2021 due to a Consul feature that didn’t work as expected. My gut with Consul is don’t use it for high-load distributed services. [1] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-...

jimmyl02 4y ago
The Roblox outage seemed like a pretty one-off instance due to a hard to catch bug. Consul still seems like a great choice and it looks like Roblox continues to use it at their scale.

mrkurt 4y ago

The design of Consul is wrong for what we need to do. Consul has been pretty good when it's running, but it's a huge pain in the ass to recover when it falls over. And when it does fall over, it's usually with no notice.

seabrookmx 4y ago

Why not both?