7 ms·

This is a good point. I wish there was more discussion around when to consider making these decisions, how to transition, what types of systems are in the middl

This is a good point. I wish there was more discussion around when to consider making these decisions, how to transition, what types of systems are in the middle ground between bedroom and facebook, etc. It seems all of the articles/posts that get are attention are the "you don't need it ever" opinions, but those are incredibly short sighted and assume that everyone is recklessly applying complex infrastructure. In general, it would be nice to have more nuanced discussions around these choices rather than the low hanging fruit that is always presented.

richardw 6y ago
No simple advice will cover all the cases but not a bad place to start: https://basecamp.com/gettingreal/04.5-scale-later https://basecamp.com/gettingreal/04.5-scale-later
cesaref 6y ago
When you are in a position to pay someone full time to look after the web infrastructure of your project, that's when you should think this stuff through. Before that, don't worry about the tech, go for what you know, and expect it to need totally re-writing at some point. Let's face it, this stuff moves quickly.
- blacktriangle 6y ago
  That's the best metric I've heard. When your Heroku bill is the same as what a full time ops person would cost, start looking to hire.
  - jacquesm 6y ago
    I would say it has to be at least double that and maybe even higher.
  - rualca 6y ago
    If you are already deploying to Heroku, you are already doing ops. It makes zero sense to presume you don't need to know what you're doing or benefit from automating processes just because you can't hire a full-time expert.
    - blacktriangle 6y ago
      True, but there's a difference between solving ops problems by throwing more money at hardware and solving ops problems by thinking smarter. That's why I think that some multiple of an FTE salary is a good metric for when you start transitioning to more complex operational setups, because it gives you a number for when the gain of that new complexity is worth the cost.
- Melkman 6y ago
  Also, when cost caused by downtime and updates and/or hardware problems exceed the cost of this full time engineer. Even then K8s is probable not the next step. Simple HA with a load balancer and seperate test and production environments plus some automation with an orchestration tool works wonders.
- marcosdumay 6y ago
  That's not good. That kind of stuff must be derived from your business plan, not from the current situation. You should know that "if I'm successful at this level, I will need this kind of infrastructure" before the problems appear, and not be caught by surprise and react in a hush. (Granted that reacting in a hush will probably happen anyway, but it shouldn't be surprising.) If your business requires enough infrastructure that this kind of tooling is a need at all, it may very well be an important variable on whether you can possibly be profitable or not.
  - marcus_holmes 6y ago
    if your business plan has this much detail, you'll spend more time updating it to reflect current reality than actually shipping features. Companies that are growing fast enough to need Kubernetes are also changing fast and learning fast.
  - lmm 6y ago
    > You should know that "if I'm successful at this level, I will need this kind of infrastructure" before the problems appear, and not be caught by surprise and react in a hush. (Granted that reacting in a hush will probably happen anyway, but it shouldn't be surprising.) What's the ROI on spending time on planning for that, given that you acknowledge that the response will be mess anyway? In my experience companies that planned for scaling don't handle it noticeably better than companies that didn't plan for it, so maybe it's better not to bother?
    - rualca 6y ago
      > What's the ROI on spending time on planning for that, given that you acknowledge that the response will be mess anyway? Let's put it this way: what's the ROI of failing to address any of the failure modes avoided or mitigated by taking the time to thing things through at design time? Are you planning on getting paid when you can't deliver value because your service is down?
      - lmm 6y ago
- marcus_holmes 6y ago
  This. When you have enough revenue to hire new dev, and infrastructure is your most pressing problem to solve with that money, hire someone who knows this stuff backwards to solve it.
jameshart 6y ago
So I agree there’s definitely a need for more writing in this space (and you’d think it might be particularly of interest to incubating startups...) My top level take on this is that the cues to take things up a level in terms of infrastructure diligence are when the risk budget you’re dealing with is no longer entirely your own. Once you have a banker, a lawyer, an insurance policy, an investor, a big customer, a government regulator, shareholders... ultimately once there’s someone you have to answer to if you haven’t done enough to protect their interests. And risk management isn’t just ‘what will happen if the server catches fire’, it’s ‘how do I guarantee the code running in production is the code I think it is?’ And ‘how do I roll back if I push a bug to production?’ And ‘how can I make sure all the current versions of the code come back up after patching my servers?’ And it turns out that things like kubernetes can help you solve some of those problems. And things like serverless can make some of those problems go away completely. But of course they come with their own problems! But when those things become important enough that you need to solve for them to answer the risk aversion of sufficient stakeholders, the cost of adopting those technologies starts to look sensible.
mumblemumble 6y ago
What I tend to see missing is clear explanations of, "What problems does this solve?" Usually you see a lot of, "What does it do?" which isn't quite the same thing. The framing is critical because it's a lot easier for people to decide whether or not they have a particular problem. If you frame it as, "Do we need this?" though, well, humans are really bad at distinguishing need from want. Take Hadoop. I wasted a lot of time and money on distributed computing once, almost entirely because of this mistake. I focused on the cool scale-out features, and of course I want to be able to scale out. Scale out is an incredibly useful thing! If, on the other hand, I had focused on the problem - "If your storage can sustain a transfer rate of X MB/s, then your theoretical minimum time to chew through Y MB of data is Y/X seconds, and, if Y is big enough, then Y/X becomes unacceptable" - then I could easily have said, "Well my Y/X is 1/10,000 of the acceptable limit and it's only growing at 20% per year, so I guess I don't really need to spend any more time thinking about this right now."
- edejong 6y ago
  We don't have data sheets, like professional electronic components have. I guess we value Turing/Church more than predictability.
- braveyellowtoad 6y ago
  Yep. All of us tech folks read sites like hackernews and read all about what the latest hot Silicon Valley tech startup is doing. Or what Uber is doing. Or what google is doing. Or what Amazon is doing. And we want to be cool as well so we jump on that new technology. Whereas for the significant majority of applications out there, older less sexy technology on modern fast computers will almost certainly be good enough and probably easier to maintain.
  - XorNot 6y ago
    Uber is one of those alleged behemoths I'm incredibly skeptical about the scaling problems of. The service they provide can't be anywhere near the Facebook/Google league of scale, nor Netflix level of data/performance demand.
yawaramin 6y ago
FTA: > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great. You have the funds to do things “right”. Your fancy zero-click continuous deployment system saves you thousands/millions a year. But at indie maker, “Hey look at this cool thing I built … please, please someone look at it (and upvote it on Product Hunt too, thx)” scale — the scale of almost every single site on the net — that 0.1% is vanishingly insignificant.
- jameshart 6y ago
  This is a really weird take on what ‘every single site on the internet’ is. Or of what proportion of the software development community is working on sites of that scale. It’s like saying ‘the vast majority of people building houses don’t need to dig foundations’ because you build lego houses, and after all the vast majority of houses are made of Lego, right?
- rualca 6y ago
  > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, I'm yet to see a company with any o line presence where downtime doesn't eat away their profits. In the very least, downtime means you're paying money to get no service. Downtime is fine for someone's blog on his personal website that he setup on a weekend while drinking beer. I'm yet to see a business state that they are ok with getting their site 404 or 500 randomly and during a whole workday, which is what 0.1% translates to.
  - yawaramin 6y ago
    Fear not, the article has you covered (emphasis mine): > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great.
    - rualca 6y ago

richardw 6y ago

No simple advice will cover all the cases but not a bad place to start: https://basecamp.com/gettingreal/04.5-scale-later https://basecamp.com/gettingreal/04.5-scale-later

cesaref 6y ago

When you are in a position to pay someone full time to look after the web infrastructure of your project, that's when you should think this stuff through. Before that, don't worry about the tech, go for what you know, and expect it to need totally re-writing at some point. Let's face it, this stuff moves quickly.

blacktriangle 6y ago
That's the best metric I've heard. When your Heroku bill is the same as what a full time ops person would cost, start looking to hire.
- jacquesm 6y ago
  I would say it has to be at least double that and maybe even higher.
- rualca 6y ago
  If you are already deploying to Heroku, you are already doing ops. It makes zero sense to presume you don't need to know what you're doing or benefit from automating processes just because you can't hire a full-time expert.
  - blacktriangle 6y ago
    True, but there's a difference between solving ops problems by throwing more money at hardware and solving ops problems by thinking smarter. That's why I think that some multiple of an FTE salary is a good metric for when you start transitioning to more complex operational setups, because it gives you a number for when the gain of that new complexity is worth the cost.
Melkman 6y ago
Also, when cost caused by downtime and updates and/or hardware problems exceed the cost of this full time engineer. Even then K8s is probable not the next step. Simple HA with a load balancer and seperate test and production environments plus some automation with an orchestration tool works wonders.
marcosdumay 6y ago
That's not good. That kind of stuff must be derived from your business plan, not from the current situation. You should know that "if I'm successful at this level, I will need this kind of infrastructure" before the problems appear, and not be caught by surprise and react in a hush. (Granted that reacting in a hush will probably happen anyway, but it shouldn't be surprising.) If your business requires enough infrastructure that this kind of tooling is a need at all, it may very well be an important variable on whether you can possibly be profitable or not.
- marcus_holmes 6y ago
  if your business plan has this much detail, you'll spend more time updating it to reflect current reality than actually shipping features. Companies that are growing fast enough to need Kubernetes are also changing fast and learning fast.
- lmm 6y ago
  > You should know that "if I'm successful at this level, I will need this kind of infrastructure" before the problems appear, and not be caught by surprise and react in a hush. (Granted that reacting in a hush will probably happen anyway, but it shouldn't be surprising.) What's the ROI on spending time on planning for that, given that you acknowledge that the response will be mess anyway? In my experience companies that planned for scaling don't handle it noticeably better than companies that didn't plan for it, so maybe it's better not to bother?
  - rualca 6y ago
    > What's the ROI on spending time on planning for that, given that you acknowledge that the response will be mess anyway? Let's put it this way: what's the ROI of failing to address any of the failure modes avoided or mitigated by taking the time to thing things through at design time? Are you planning on getting paid when you can't deliver value because your service is down?
    - lmm 6y ago
marcus_holmes 6y ago
This. When you have enough revenue to hire new dev, and infrastructure is your most pressing problem to solve with that money, hire someone who knows this stuff backwards to solve it.

jameshart 6y ago

So I agree there’s definitely a need for more writing in this space (and you’d think it might be particularly of interest to incubating startups...) My top level take on this is that the cues to take things up a level in terms of infrastructure diligence are when the risk budget you’re dealing with is no longer entirely your own. Once you have a banker, a lawyer, an insurance policy, an investor, a big customer, a government regulator, shareholders... ultimately once there’s someone you have to answer to if you haven’t done enough to protect their interests. And risk management isn’t just ‘what will happen if the server catches fire’, it’s ‘how do I guarantee the code running in production is the code I think it is?’ And ‘how do I roll back if I push a bug to production?’ And ‘how can I make sure all the current versions of the code come back up after patching my servers?’ And it turns out that things like kubernetes can help you solve some of those problems. And things like serverless can make some of those problems go away completely. But of course they come with their own problems! But when those things become important enough that you need to solve for them to answer the risk aversion of sufficient stakeholders, the cost of adopting those technologies starts to look sensible.

mumblemumble 6y ago

What I tend to see missing is clear explanations of, "What problems does this solve?" Usually you see a lot of, "What does it do?" which isn't quite the same thing. The framing is critical because it's a lot easier for people to decide whether or not they have a particular problem. If you frame it as, "Do we need this?" though, well, humans are really bad at distinguishing need from want. Take Hadoop. I wasted a lot of time and money on distributed computing once, almost entirely because of this mistake. I focused on the cool scale-out features, and of course I want to be able to scale out. Scale out is an incredibly useful thing! If, on the other hand, I had focused on the problem - "If your storage can sustain a transfer rate of X MB/s, then your theoretical minimum time to chew through Y MB of data is Y/X seconds, and, if Y is big enough, then Y/X becomes unacceptable" - then I could easily have said, "Well my Y/X is 1/10,000 of the acceptable limit and it's only growing at 20% per year, so I guess I don't really need to spend any more time thinking about this right now."

edejong 6y ago
We don't have data sheets, like professional electronic components have. I guess we value Turing/Church more than predictability.
braveyellowtoad 6y ago
Yep. All of us tech folks read sites like hackernews and read all about what the latest hot Silicon Valley tech startup is doing. Or what Uber is doing. Or what google is doing. Or what Amazon is doing. And we want to be cool as well so we jump on that new technology. Whereas for the significant majority of applications out there, older less sexy technology on modern fast computers will almost certainly be good enough and probably easier to maintain.
- XorNot 6y ago
  Uber is one of those alleged behemoths I'm incredibly skeptical about the scaling problems of. The service they provide can't be anywhere near the Facebook/Google league of scale, nor Netflix level of data/performance demand.

yawaramin 6y ago

FTA: > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great. You have the funds to do things “right”. Your fancy zero-click continuous deployment system saves you thousands/millions a year. But at indie maker, “Hey look at this cool thing I built … please, please someone look at it (and upvote it on Product Hunt too, thx)” scale — the scale of almost every single site on the net — that 0.1% is vanishingly insignificant.

jameshart 6y ago
This is a really weird take on what ‘every single site on the internet’ is. Or of what proportion of the software development community is working on sites of that scale. It’s like saying ‘the vast majority of people building houses don’t need to dig foundations’ because you build lego houses, and after all the vast majority of houses are made of Lego, right?
rualca 6y ago
> Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, I'm yet to see a company with any o line presence where downtime doesn't eat away their profits. In the very least, downtime means you're paying money to get no service. Downtime is fine for someone's blog on his personal website that he setup on a weekend while drinking beer. I'm yet to see a business state that they are ok with getting their site 404 or 500 randomly and during a whole workday, which is what 0.1% translates to.
- yawaramin 6y ago
  Fear not, the article has you covered (emphasis mine): > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great.
  - rualca 6y ago