3 ms·
Well, why did you do that? If you’d asked them whether this was a supported configuration or intended purpose, they’d have said no; and anyone who had experienc
by otterley 4y ago
Well, why did you do that? If you’d asked them whether this was a supported configuration or intended purpose, they’d have said no; and anyone who had experience deploying Consul at large scale would have told you the same.
There is truly no compression algorithm for experience.
- mixmastamyk 4y agoI don't think he personally designed the first implementation. But in any case, understanding of complex topics comes in waves. Many times I've had to read all the docs then use a system for several months before the epiphany hits me.
- otterley 4y agoI also think there’s this tendency in the industry to want to solve problems on your own without the help from outsiders, even if they know the problem space better than you do, and even if they’d gladly help (often for free) if asked. It’s especially worrisome when it’s powering a key workload that is essential to the functioning of your business. Sometimes it’s because you might not know whom to consult or recruit, but in this case, the vendor was known.
- JeremyNT 4y agoThis is especially true for scaling. A solution that works great for your current deployment may be completely unworkable for 2x your current deployment. You just won't know until you fall off the cliff. The armchair quarterback can opine that you should have just hired experts in XYZ domains from the start to design robust systems that can scale to arbitrary sizes, but most orgs don't need to scale to arbitrary sizes so this is highly likely to be wasted effort.
- otterley 4y agoWhile I largely agree with you, this isn’t one of those cases. If Fly wasn’t supposed to scale in due course to this size, it probably wouldn’t have been funded. If your business model is predicated on you scaling, yes, you should hire appropriately in anticipation of that. Besides, I’m not even necessarily talking about hiring here - even consulting would have been sufficient to avoid this catastrophe.
- mixmastamyk 4y agoYes, although it's rarely possible to know which bottlenecks will hurt the most up front. Unless you've done the same thing before, which is not the case with anyone pushing boundaries. Basically this is an argument around so-called premature optimization. Good to have issues now while it is mostly enthusiasts that are the customers. Guessing that this bump will be forgotten in five years? And not like AWS et al don't have outages occasionally that they learn from.
- otterley 4y agoConsul has been around for close to 9 years now, and people have in fact tried to use Consul in the very same way Fly did, in many different business and industries, with similarly failing outcomes. Hashicorp knows this and almost certainly would have counseled against it if asked.
- mixmastamyk 4y agoInsert Donald Rumsfeld quote about un/known un/knowns.
- bovermyer 4y agoThis feels unnecessarily antagonistic. "If you were experienced, you would have made the right decision, _obviously_." Did Fly.io kick your puppy or something?
- otterley 4y agoI can see how it would be interpreted that way, and I apologize if it came across that way, but it wasn’t my intent. See my other comment below. What I’m really saying is that we need to be better about engaging subject matter experts early on when we are selecting technologies to power core business functions; and I think it’s a good illustration of why we need to continue to hire experienced people at startups.
- bovermyer 4y agoThat's a fair point... but at the same time, we shouldn't hold off on starting something just because we don't have perfect information.