6 ms·

>is obvious until you realize it means, for example, allowing idle time on incredibly expensive machines to favor responsiveness, which just seems wasteful. We

>is obvious until you realize it means, for example, allowing idle time on incredibly expensive machines to favor responsiveness, which just seems wasteful. Weird how things that seem to make sense in one context seem to make no sense in another context. If you told me a factory runs their widget making machine at 70% capacity in case someone comes along with an order for a different widget or twice as many widgets, at first glance think that's a bad idea. If your customers can keep your widget machine 100% full, using only part of the machine for the chance that something new will come along seems wasteful. And through cultural osmosis the idea of not letting your hardware sit idle is exactly the sort of thing that feels right. And yet, we do this all the time in IT. If you instead of a widget machine told me that you run your web server at 100% capacity all the time, I'd tell you that's also a terrible idea. If you're running at 100% capacity and have no spare headroom, you can't serve more users if one of them sends more requests than normal. Even though intuitively we know that a machine sitting idle is a "waste" of compute power, we also know that we need capacity in reserve because demand isn't constant. No one sizes (or should size) their servers for 100% utilization. Even when you have something like a container cluster, you don't target your containers to 100% utilization, if for no other reason than you need headroom while the extra containers spin up. Odd that without thinking that through, I wouldn't have applied the same idea to manufacturing machinery.

makeitdouble 2y ago
I see the parallel you're drawing but even the core idea is I think different enough to be worse dissecting. In manufacturing, you keep spare capacity to allow for more lucrative orders to come in. If you don't expect any, you run at 100%. For instance when Apple pays TSMC all the money in the world to produce the next iPhone chip, they won't be running that line at 70%, the full capacity is reserved. Or if you're a bakery, you won't keep two or three cake cooking spots just on case someone comes in witb an extraordinary order, you won't make enough on that to cover the lost opportunity. We run our servers at 70% or even 50% capacity because we don't have control on what that capacity will be used for, as external events happen all the time. A manufacturers receiving a spike of extra orders can just refuse them and go on with their day. Our servers getting hit with 10x the demand requires efforts and measures to protect the servers and current traffic. Factories want to optimize for efficiency, server farms want to pay for more reactivity, that's the nature of the business.
- tpmoney 2y ago
  > Or if you're a bakery, you won't keep two or three cake cooking spots just on case someone comes in witb an extraordinary order, you won't make enough on that to cover the lost opportunity. I think it's always worth thinking about what you can leave slack / idle space in. For example, you might not keep multiple stations free, but you might invest in a larger oven than you need to make the cakes you currently make. Or you might invest in more bakery space than you need, including extra workspace than you can utilize at 100%. Not because you necessarily anticipate higher demand, but because you might get a customer that's asking for a cake bigger than your standard. Or because you might have a customer placing a large order and need some extra room to spread out more, or to have a temporary helper be able to do some small part of the job even if they can't use the space as a full station. But also idleness might look like "you don't spend all of your time baking orders for customers". If you never build in slack for creating, experimenting and learning, you'll fall behind your competition, or stagnate if your design and art is a selling point.
- kookybakker 2y ago
  I think even for a company like TSMC these ideas are important to understand. To give you an example TSMC might have a factory with 10 expensive EUV lithography tools, each capable of processing 100 wafers per hour. Then they have 4 ovens, each able to bake batches of 500 wafers per hour. TSMC could improve efficiency by reducing the number of ovens, because they are running only at 50% capacity. But compared to the cost of the EUV tools, the ovens are very cheap. They need to be able to produce at full capacity, even when some ovens breakdown, because stopping the EUV tools because you don't have enough ovens would be much more expensive then operating with spare capacity.
- geon 2y ago
  Even for a server farm, you can prioritize the web traffic and still use the excess capacity for CI or whatever.
DaoVeles 2y ago
This is a very key insight many need to be aware of. The thing that can be sacrificed in order to obtain efficiency is resilience. To master the bend not break model. You can make a bridge that can handle a 10 ton load for half the material of one that can take 20 tons. 99% of the time this isn't an issue but that outlier case of a 18 ton truck can be disastrous. This is why power cables have sag in them, in case there is an extreme cold snap. Why trees sway and bend with the wind so that anything but the most extreme evens do not break them; with that analogy, grass is much weaker but could handle even higher winds. The ridged are brittle. I'm not saying to not strive for efficiency but you also have to allow those efficiency gains to provide some slack in the system. Where I work, there is a definite busy season. So for most of the year, we operate at about 70% utilization and it works out great. Most people are not stressed at all. It means that when those 2 months of the year when it is all hands on deck, everyone is in peak condition to face it head on. In my previous job in manufacturing, efficiency was praised over everything else, it was 100% utilization all of the time. So when the COVID rush came, it practically broke the business. After a year of those unrelenting pace, we started to bleed out talent. Over the next 6 months, they lost all the highest talent. A year later from those I still spoke with, they said they lost about two thirds of their business over the next 12 months, they are now on the edge of collapse. Slack allows a bend, pure efficiency can lead to a break. There is a fine line between those two that is very difficult to achieve.
citizenpaul 2y ago
> using only part of the machine for the chance that something new will come along seems wasteful. Because it is. My brother works in industrial manufacturing machinery supplies. I can assure you the overwhelming majority of manufacturing machines on the planet are not only run constantly but as near to 99.999% as possible. So much that they are even loath to turn them off for critical maintenance rather preferring to let the machine break down so they don't get blamed for being the person to "ruin productivity" This book sounds like one of those flights of fancy armchair generals are so found of going on. Perhaps it works in small boutique shops making specialized orders but that is a slim minority of the overall manufacturing base. I could see why the advice would appeal to HN readers.
- llamaimperative 2y ago
  Yeah it mostly only works for small boutique shops like the Toyota Production System or Ford’s manufacturing line. And yes, a lot of manufacturing doesn’t behave this way. That’s the “counter” part of “counter-intuitive” revealing itself. This comment is yet another of these excellent cases in point! You really don’t see how “they’re afraid to turn them off even for critical maintenance” might be actually suboptimal behavior in the long run?
  - jiggawatts 2y ago
    One of the most insightful things I heard someone say at Toyota (in an interview) was that they replace their tools (drill bits and the like) at 80% wear instead of letting them get to 100% and break. Why waste that 20%? Because if the tool breaks and scratches a $200K Lexus, then that might be a $20K fix, or possibly even starting from scratch with a new body! Is that worth risking for a $5 drill bit they buy in boxes of 1,000 at a time? No. Then the interview switched to some guy in America looking miserable complaining how his bosses made him use every tool until breaking point. He listed a litany of faults this caused, like off-centre holes, distorted panels, etc... And you wonder why Tesla panels have misaligned gaps. Or why rain water leaks into a "luxury" American vehicle!
    - 2y ago

makeitdouble 2y ago

I see the parallel you're drawing but even the core idea is I think different enough to be worse dissecting. In manufacturing, you keep spare capacity to allow for more lucrative orders to come in. If you don't expect any, you run at 100%. For instance when Apple pays TSMC all the money in the world to produce the next iPhone chip, they won't be running that line at 70%, the full capacity is reserved. Or if you're a bakery, you won't keep two or three cake cooking spots just on case someone comes in witb an extraordinary order, you won't make enough on that to cover the lost opportunity. We run our servers at 70% or even 50% capacity because we don't have control on what that capacity will be used for, as external events happen all the time. A manufacturers receiving a spike of extra orders can just refuse them and go on with their day. Our servers getting hit with 10x the demand requires efforts and measures to protect the servers and current traffic. Factories want to optimize for efficiency, server farms want to pay for more reactivity, that's the nature of the business.

tpmoney 2y ago
> Or if you're a bakery, you won't keep two or three cake cooking spots just on case someone comes in witb an extraordinary order, you won't make enough on that to cover the lost opportunity. I think it's always worth thinking about what you can leave slack / idle space in. For example, you might not keep multiple stations free, but you might invest in a larger oven than you need to make the cakes you currently make. Or you might invest in more bakery space than you need, including extra workspace than you can utilize at 100%. Not because you necessarily anticipate higher demand, but because you might get a customer that's asking for a cake bigger than your standard. Or because you might have a customer placing a large order and need some extra room to spread out more, or to have a temporary helper be able to do some small part of the job even if they can't use the space as a full station. But also idleness might look like "you don't spend all of your time baking orders for customers". If you never build in slack for creating, experimenting and learning, you'll fall behind your competition, or stagnate if your design and art is a selling point.
kookybakker 2y ago
I think even for a company like TSMC these ideas are important to understand. To give you an example TSMC might have a factory with 10 expensive EUV lithography tools, each capable of processing 100 wafers per hour. Then they have 4 ovens, each able to bake batches of 500 wafers per hour. TSMC could improve efficiency by reducing the number of ovens, because they are running only at 50% capacity. But compared to the cost of the EUV tools, the ovens are very cheap. They need to be able to produce at full capacity, even when some ovens breakdown, because stopping the EUV tools because you don't have enough ovens would be much more expensive then operating with spare capacity.
geon 2y ago
Even for a server farm, you can prioritize the web traffic and still use the excess capacity for CI or whatever.

DaoVeles 2y ago

This is a very key insight many need to be aware of. The thing that can be sacrificed in order to obtain efficiency is resilience. To master the bend not break model. You can make a bridge that can handle a 10 ton load for half the material of one that can take 20 tons. 99% of the time this isn't an issue but that outlier case of a 18 ton truck can be disastrous. This is why power cables have sag in them, in case there is an extreme cold snap. Why trees sway and bend with the wind so that anything but the most extreme evens do not break them; with that analogy, grass is much weaker but could handle even higher winds. The ridged are brittle. I'm not saying to not strive for efficiency but you also have to allow those efficiency gains to provide some slack in the system. Where I work, there is a definite busy season. So for most of the year, we operate at about 70% utilization and it works out great. Most people are not stressed at all. It means that when those 2 months of the year when it is all hands on deck, everyone is in peak condition to face it head on. In my previous job in manufacturing, efficiency was praised over everything else, it was 100% utilization all of the time. So when the COVID rush came, it practically broke the business. After a year of those unrelenting pace, we started to bleed out talent. Over the next 6 months, they lost all the highest talent. A year later from those I still spoke with, they said they lost about two thirds of their business over the next 12 months, they are now on the edge of collapse. Slack allows a bend, pure efficiency can lead to a break. There is a fine line between those two that is very difficult to achieve.

citizenpaul 2y ago

> using only part of the machine for the chance that something new will come along seems wasteful. Because it is. My brother works in industrial manufacturing machinery supplies. I can assure you the overwhelming majority of manufacturing machines on the planet are not only run constantly but as near to 99.999% as possible. So much that they are even loath to turn them off for critical maintenance rather preferring to let the machine break down so they don't get blamed for being the person to "ruin productivity" This book sounds like one of those flights of fancy armchair generals are so found of going on. Perhaps it works in small boutique shops making specialized orders but that is a slim minority of the overall manufacturing base. I could see why the advice would appeal to HN readers.

llamaimperative 2y ago
Yeah it mostly only works for small boutique shops like the Toyota Production System or Ford’s manufacturing line. And yes, a lot of manufacturing doesn’t behave this way. That’s the “counter” part of “counter-intuitive” revealing itself. This comment is yet another of these excellent cases in point! You really don’t see how “they’re afraid to turn them off even for critical maintenance” might be actually suboptimal behavior in the long run?
- jiggawatts 2y ago
  One of the most insightful things I heard someone say at Toyota (in an interview) was that they replace their tools (drill bits and the like) at 80% wear instead of letting them get to 100% and break. Why waste that 20%? Because if the tool breaks and scratches a $200K Lexus, then that might be a $20K fix, or possibly even starting from scratch with a new body! Is that worth risking for a $5 drill bit they buy in boxes of 1,000 at a time? No. Then the interview switched to some guy in America looking miserable complaining how his bosses made him use every tool until breaking point. He listed a litany of faults this caused, like off-centre holes, distorted panels, etc... And you wonder why Tesla panels have misaligned gaps. Or why rain water leaks into a "luxury" American vehicle!
  - 2y ago