7 ms·

I think you need to work at companies that are large enough to allow for people to dedicate their time to this level of analysis. Smaller companies are too busy

I think you need to work at companies that are large enough to allow for people to dedicate their time to this level of analysis. Smaller companies are too busy to have staff focussing on things like this.

xxpor 4y ago
Seems like poor analysis on their part then. If they're leaving 3.5x perf on the floor, they're not spending their money very well.
- loeg 4y ago
  A lot of businesses reasonably use Python and Ruby, for example.
  - snovv_crash 4y ago
    If you can run 1 server instead of 3 though... that means no ansible, no kubernetes, no wild deployment strategies, 1/3 the likelihood of hardware failure, etc
    - maccard 4y ago
      This isn't true, at all. We are not "high availability" by any means but running on one server has significant risks, _especially_ if you don't use some form of automated provisioning. A destructive app logic bug is far more likely than a hardware failure, and in both cases the impact on your service is significant if you have one instance, but likely manageable if you have more than one.
    - heurisko 4y ago
      Why wouldn't you use ansible to manage one server? It's a great tool with a low learning curve that just requires SSH.
- magicalhippo 4y ago
  Leaving 3.5x perf of what on the floor though? We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms, or if the integration service that processes a few dozen files per day spends 5 seconds instead of 1 second. I'll easily trade 5x performance in 99% of my code if it makes me produce 5x more features, because most of the time my code runs in just a few milliseconds at a time anyway. Of course, I'm weary of big-Oh, that one will bit ya. But a mere 3.5x is almost never worth chasing for us.
  - conradev 4y ago
    Netflix is not doing this work to improve user latency, they are doing this work to minimize cost: > At Netflix, we periodically reevaluate our workloads to optimize utilization of available capacity. The idea being that if you pay for fewer servers you spend less money
    - abrookewood 4y ago
      And at their scale, this makes sense. No one is going to do this work if if means they run 2 servers instead of 6.
  - anonymoushn 4y ago
    > We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. If you try to be good at waiting on many things, you can use one machine instead of a hundred > None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms They absolutely will, what
    - magicalhippo 4y ago
      > If you try to be good at waiting on many things, you can use one machine instead of a hundred We can't run on one, because customers run our application on-prem. > They absolutely will, what How can you be so sure?
amscanne 4y ago
It’s not “too busy”. It’s that unless you have significant scale, the return for improvements like this are often not worth the engineering time. At companies with massive scale (read: large enough), even tiny improvements can pay for themselves.
- mmis1000 4y ago
  It's probably cheaper for a small company to just add another machine instead of hire a whole team to do these.
  - abrookewood 4y ago
    I think you and the parent comment are both correct. Engineers who are skilled enough to do this type of analysis are expensive and another VM/Instance is a few hundred a month at most.
    - jjav 4y ago
      At the very small end of the scale this is very true. It doesn't take too much traffic volume to make it worth it though. It's just difficult to make anyone care. It doesn't take a particularly huge amount of traffic to make it worth spending a few weeks of engineering time to save $50K+ on annual hosting costs.
      - amscanne 4y ago
        I think you’re considering only literal cost, and not the opportunity cost (which is typically much higher, and what you’re using to make investment decisions). Suppose you’ve an engineer and they’re paid $100k. Now you’d be tempted to say that anything that takes less than six months and saves more than $50k is worth it. But that’s not true at all. For one, that engineer is actually worth much more than $100k/yr to the company; it costs a lot more to hire and keep that person busy (recruiting, training, an office, a manager, product managers, project managers, etc.). But more importantly: what are the other things they could be doing? Small companies are rarely thinking about micro-efficiency because they are trying to change and grow their products. If this engineer is able to build a feature that will help them grow X% faster, that can have a massive multiplier effect on their prospects (and valuation, for which that $50k saving makes zero difference). Those are the things you’re comparing against, which is why the opportunity cost bar for pure $$ saving improvements is often much higher than it seems.
fnordpiglet 4y ago
I’ve been at the largest companies on earth for about 20 years. They are precisely who can’t countenance such work.
- abrookewood 4y ago
  Most of the blog posts I see that are this detailed are usually from larger companies.
  - fnordpiglet 4y ago
    I don’t think it’s related to size but rather corporate culture, and what behavior leads to profitability. Netflix is all about reducing cost in their technical infrastructure while providing a highly consistent high bandwidth relatively low latency experience. Exxon Mobil couldn’t care less. There are small pockets of Amazon that care, but most of Amazon is product management driven towards a business goal where costs and performance are only relevant when it interferes with the product goal. It’s less about size and more about priority for achieving the core business objectives. Companies that need this behavior but don’t prioritize it will sooner or later be overtaken by a Netflix. Companies that do not need this behavior but do prioritize it will be overtaken by a product focused competitor while the engineers and flipping bits to optimize marginal compute utilization.
snovv_crash 4y ago
You need more staff to manage a cluster than a single machine though. Over the course of 6 months you'll probably spend more time on maintaining the cluster (with all its overhead) than doing an optimization analysis to net you 3x performance.

xxpor 4y ago

Seems like poor analysis on their part then. If they're leaving 3.5x perf on the floor, they're not spending their money very well.

loeg 4y ago
A lot of businesses reasonably use Python and Ruby, for example.
- snovv_crash 4y ago
  If you can run 1 server instead of 3 though... that means no ansible, no kubernetes, no wild deployment strategies, 1/3 the likelihood of hardware failure, etc
  - maccard 4y ago
    This isn't true, at all. We are not "high availability" by any means but running on one server has significant risks, _especially_ if you don't use some form of automated provisioning. A destructive app logic bug is far more likely than a hardware failure, and in both cases the impact on your service is significant if you have one instance, but likely manageable if you have more than one.
  - heurisko 4y ago
    Why wouldn't you use ansible to manage one server? It's a great tool with a low learning curve that just requires SSH.
magicalhippo 4y ago
Leaving 3.5x perf of what on the floor though? We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms, or if the integration service that processes a few dozen files per day spends 5 seconds instead of 1 second. I'll easily trade 5x performance in 99% of my code if it makes me produce 5x more features, because most of the time my code runs in just a few milliseconds at a time anyway. Of course, I'm weary of big-Oh, that one will bit ya. But a mere 3.5x is almost never worth chasing for us.
- conradev 4y ago
  Netflix is not doing this work to improve user latency, they are doing this work to minimize cost: > At Netflix, we periodically reevaluate our workloads to optimize utilization of available capacity. The idea being that if you pay for fewer servers you spend less money
  - abrookewood 4y ago
    And at their scale, this makes sense. No one is going to do this work if if means they run 2 servers instead of 6.
- anonymoushn 4y ago
  > We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. If you try to be good at waiting on many things, you can use one machine instead of a hundred > None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms They absolutely will, what
  - magicalhippo 4y ago
    > If you try to be good at waiting on many things, you can use one machine instead of a hundred We can't run on one, because customers run our application on-prem. > They absolutely will, what How can you be so sure?

amscanne 4y ago

It’s not “too busy”. It’s that unless you have significant scale, the return for improvements like this are often not worth the engineering time. At companies with massive scale (read: large enough), even tiny improvements can pay for themselves.

mmis1000 4y ago
It's probably cheaper for a small company to just add another machine instead of hire a whole team to do these.
- abrookewood 4y ago
  I think you and the parent comment are both correct. Engineers who are skilled enough to do this type of analysis are expensive and another VM/Instance is a few hundred a month at most.
  - jjav 4y ago
    At the very small end of the scale this is very true. It doesn't take too much traffic volume to make it worth it though. It's just difficult to make anyone care. It doesn't take a particularly huge amount of traffic to make it worth spending a few weeks of engineering time to save $50K+ on annual hosting costs.
    - amscanne 4y ago
      I think you’re considering only literal cost, and not the opportunity cost (which is typically much higher, and what you’re using to make investment decisions). Suppose you’ve an engineer and they’re paid $100k. Now you’d be tempted to say that anything that takes less than six months and saves more than $50k is worth it. But that’s not true at all. For one, that engineer is actually worth much more than $100k/yr to the company; it costs a lot more to hire and keep that person busy (recruiting, training, an office, a manager, product managers, project managers, etc.). But more importantly: what are the other things they could be doing? Small companies are rarely thinking about micro-efficiency because they are trying to change and grow their products. If this engineer is able to build a feature that will help them grow X% faster, that can have a massive multiplier effect on their prospects (and valuation, for which that $50k saving makes zero difference). Those are the things you’re comparing against, which is why the opportunity cost bar for pure $$ saving improvements is often much higher than it seems.

fnordpiglet 4y ago

I’ve been at the largest companies on earth for about 20 years. They are precisely who can’t countenance such work.

abrookewood 4y ago
Most of the blog posts I see that are this detailed are usually from larger companies.
- fnordpiglet 4y ago
  I don’t think it’s related to size but rather corporate culture, and what behavior leads to profitability. Netflix is all about reducing cost in their technical infrastructure while providing a highly consistent high bandwidth relatively low latency experience. Exxon Mobil couldn’t care less. There are small pockets of Amazon that care, but most of Amazon is product management driven towards a business goal where costs and performance are only relevant when it interferes with the product goal. It’s less about size and more about priority for achieving the core business objectives. Companies that need this behavior but don’t prioritize it will sooner or later be overtaken by a Netflix. Companies that do not need this behavior but do prioritize it will be overtaken by a product focused competitor while the engineers and flipping bits to optimize marginal compute utilization.

snovv_crash 4y ago

You need more staff to manage a cluster than a single machine though. Over the course of 6 months you'll probably spend more time on maintaining the cluster (with all its overhead) than doing an optimization analysis to net you 3x performance.