4 ms·

> we’re good at consul Thank god someone is. I’ve lost more of my life to consul partition failures that’s any other part of the nutech stack.

by asguy 4y ago

madeofpalk 4y ago
They've had their fair share of pains with it, but they still seem pretty happy with it https://fly.io/blog/a-foolish-consistency/ https://fly.io/blog/a-foolish-consistency/
- tptacek 4y ago
  Happy is a word. There are lots of words. I like words! You could be creative about what word could take the place of "happy" in that sentence, and probably still be correct. Get weird with it! Maybe "sanguine" would work. "Engaged". The reality is: we've got a fair bit of experience with Consul at this point, we respect the hell out of it for the problems it was designed to solve, and we're unlikely to stretch it any further than we've already stretched it. Distributed lock service for Postgres clusters? Sure. Source of truth for all our app state? We've built our own thingy ("corrosion", a Rust distribute state system) to phase Consul out with. I'll get Jerome to say things about it.
  - subarctic 4y ago
    >Sure. Source of truth for all our app state? We've built our own thingy ("corrosion", a Rust distribute state system) to phase Consul out with. I'll get Jerome to say things about it. Looking forward to the blog post!
  - madeofpalk 4y ago
    Sorry yes. 'Invested' was more along what I was thinking :)
WFHRenaissance 4y ago
What else would you include in the nutech stack? HashiCorp products in general?
atonse 4y ago
Yup our experience with brittleness of consul and nomad soured me on the HashiCorp stack even though I was very excited about it. We went from buying enterprise licenses to throwing it all out without ever pushing to production in the span of a year. This stuff, for all the hype about raft and things, is brittle enough and requires enough special attention. Which is why I’d gladly rather have it be fly.io’s problem.
- tptacek 4y ago
  For what it's worth, if I was deploying a bounded set of applications or services in a small number of geographies and couldn't use Fly.io for some reason, I wouldn't hesitate to use Nomad. Nomad is pretty great; it's like Flask to K8s's Django.
  - atonse 4y ago
    I did love the simplicity of nomad. And in general, nomad worked pretty well for us but our consul cluster kept mysteriously failing. I think that caused our nomad cluster to fail because it was backed by Consul. The one complaint I did have about nomad (same as consul) was that the recovery process was manual where you had to manually generate a peers.json. I was shocked when I saw that. Truly one of the "finding how the sausage is made" moments even though I've managed linux servers for two decades – I always assumed it would use zeroconf/bonjour/multicast DNS (remember cloud auto-join?) or something similarly elegant to auto discover other nodes in the network and just reconnect and rebuild a cluster. I mean what's the point of all this stuff if it can't be used to recover a cluster and just Do The Right Thing™? The shiny new experience is stellar (like sales, or setting up a new cluster), but the flip side (when things go wrong) is a mess. That's why we eventually said "nope!" to all the custom stuff and went with boring, plain vanilla ECS, which is itself too much now that we've started using fly. Don't ever want to even think about having to hand-write a peers.json file to recover a cluster, boot things up, and pray to the ancient gods that it works. We don't have time for that nonsense. Please, take my money, Fly/Render/everyone else. Your costs are a margin of error compared to what I had to pay a devops person to build our own stack. (I'm not even exaggerating. It was six figures. DevOps people are worth every penny but they cost many, many pennies.) Ultimately, we never used the infra. I want to focus on building solutions for my customers and not fiddling with weird server stuff.

> we’re good at consul Thank god someone is. I’ve lost more of my life to consul partition failures that’s any other part of the nutech stack.

by asguy 4y ago

> we’re good at consul Thank god someone is. I’ve lost more of my life to consul partition failures that’s any other part of the nutech stack.

madeofpalk 4y ago

They've had their fair share of pains with it, but they still seem pretty happy with it https://fly.io/blog/a-foolish-consistency/ https://fly.io/blog/a-foolish-consistency/

tptacek 4y ago
Happy is a word. There are lots of words. I like words! You could be creative about what word could take the place of "happy" in that sentence, and probably still be correct. Get weird with it! Maybe "sanguine" would work. "Engaged". The reality is: we've got a fair bit of experience with Consul at this point, we respect the hell out of it for the problems it was designed to solve, and we're unlikely to stretch it any further than we've already stretched it. Distributed lock service for Postgres clusters? Sure. Source of truth for all our app state? We've built our own thingy ("corrosion", a Rust distribute state system) to phase Consul out with. I'll get Jerome to say things about it.
- subarctic 4y ago
  >Sure. Source of truth for all our app state? We've built our own thingy ("corrosion", a Rust distribute state system) to phase Consul out with. I'll get Jerome to say things about it. Looking forward to the blog post!
- madeofpalk 4y ago
  Sorry yes. 'Invested' was more along what I was thinking :)

WFHRenaissance 4y ago

What else would you include in the nutech stack? HashiCorp products in general?

atonse 4y ago

Yup our experience with brittleness of consul and nomad soured me on the HashiCorp stack even though I was very excited about it. We went from buying enterprise licenses to throwing it all out without ever pushing to production in the span of a year. This stuff, for all the hype about raft and things, is brittle enough and requires enough special attention. Which is why I’d gladly rather have it be fly.io’s problem.

tptacek 4y ago
For what it's worth, if I was deploying a bounded set of applications or services in a small number of geographies and couldn't use Fly.io for some reason, I wouldn't hesitate to use Nomad. Nomad is pretty great; it's like Flask to K8s's Django.
- atonse 4y ago
  I did love the simplicity of nomad. And in general, nomad worked pretty well for us but our consul cluster kept mysteriously failing. I think that caused our nomad cluster to fail because it was backed by Consul. The one complaint I did have about nomad (same as consul) was that the recovery process was manual where you had to manually generate a peers.json. I was shocked when I saw that. Truly one of the "finding how the sausage is made" moments even though I've managed linux servers for two decades – I always assumed it would use zeroconf/bonjour/multicast DNS (remember cloud auto-join?) or something similarly elegant to auto discover other nodes in the network and just reconnect and rebuild a cluster. I mean what's the point of all this stuff if it can't be used to recover a cluster and just Do The Right Thing™? The shiny new experience is stellar (like sales, or setting up a new cluster), but the flip side (when things go wrong) is a mess. That's why we eventually said "nope!" to all the custom stuff and went with boring, plain vanilla ECS, which is itself too much now that we've started using fly. Don't ever want to even think about having to hand-write a peers.json file to recover a cluster, boot things up, and pray to the ancient gods that it works. We don't have time for that nonsense. Please, take my money, Fly/Render/everyone else. Your costs are a margin of error compared to what I had to pay a devops person to build our own stack. (I'm not even exaggerating. It was six figures. DevOps people are worth every penny but they cost many, many pennies.) Ultimately, we never used the infra. I want to focus on building solutions for my customers and not fiddling with weird server stuff.