5 ms·

These posts both make me violently aroused as well as profoundly depressed basically every employer I’ve ever had in 30 years doesn’t value such analysis. This

These posts both make me violently aroused as well as profoundly depressed basically every employer I’ve ever had in 30 years doesn’t value such analysis. This is what engineering looks like, and it’s something no product manager can ever comprehend.

skellington 4y ago
Yup, very few workplaces would know how to reward people with this skill level.
- fnordpiglet 4y ago
  Almost every engineer I’ve worked with have this skill level, but they’re not given the leeway, direction, or advice to get to the conclusion. Netflix must have an amazing culture in these areas.
  - ilyt 4y ago
    Really that just need manager that presents the results to higher-ups as raw cash savings. "My team makes your team apps and make them run cheaper"
  - bitcharmer 4y ago
    I've been a high performance systems engineer for a number of investment banks over the last 10+ years and people with deep knowledge like that are quite rare; that's why we get paid handsomely. Mind sharing what field you work in that gathers so much talent like that?
    - fnordpiglet 4y ago
      Knowledge != skill. With the skill you can acquire the knowledge given the right conditions and motivation. I would say the vast majority of engineers I work with have never had the need or opportunity to invest the time or experiential opportunities to acquire the knowledge required - but they COULD. I actually find that quite sad, and hence my depression I noted. Nowadays this is extending all the way through school. Machine architecture and low level machine microcode etc isn’t universally taught in CS programs any more. I’ve worked in all the industries, including a long time in yours. We probably even overlapped or maybe worked together!
- saagarjha 4y ago
  You have to know how to justify your work. Places like Netflix have teams like these because they literally print money, with each engineer often saving the company millions if not tens of millions of dollars annually. It’s hard to argue with results.
- lmm 4y ago
  Very few workplaces value this skill because very few workplaces have a use for it. One service now runs 3x faster? You reduced the AWS bill for this one service by 2/3? That's not going to change the bottom line enough for anyone to care about, unless you're, well, Netflix.
abrookewood 4y ago
I think you need to work at companies that are large enough to allow for people to dedicate their time to this level of analysis. Smaller companies are too busy to have staff focussing on things like this.
- xxpor 4y ago
  Seems like poor analysis on their part then. If they're leaving 3.5x perf on the floor, they're not spending their money very well.
  - loeg 4y ago
    A lot of businesses reasonably use Python and Ruby, for example.
    - snovv_crash 4y ago
      If you can run 1 server instead of 3 though... that means no ansible, no kubernetes, no wild deployment strategies, 1/3 the likelihood of hardware failure, etc
      - maccard 4y ago
        This isn't true, at all. We are not "high availability" by any means but running on one server has significant risks, _especially_ if you don't use some form of automated provisioning. A destructive app logic bug is far more likely than a hardware failure, and in both cases the impact on your service is significant if you have one instance, but likely manageable if you have more than one.
      - heurisko 4y ago
        Why wouldn't you use ansible to manage one server? It's a great tool with a low learning curve that just requires SSH.
  - magicalhippo 4y ago
    Leaving 3.5x perf of what on the floor though? We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms, or if the integration service that processes a few dozen files per day spends 5 seconds instead of 1 second. I'll easily trade 5x performance in 99% of my code if it makes me produce 5x more features, because most of the time my code runs in just a few milliseconds at a time anyway. Of course, I'm weary of big-Oh, that one will bit ya. But a mere 3.5x is almost never worth chasing for us.
newZWhoDis 4y ago
Fuck “project managers”. The second you are subjected to one as an engineer you are being told you’re in the wrong place. I don’t care what the stock options are, I don’t care what the free morning deviled eggs are like, that place is cancer and you need to escape it.
harha 4y ago
PM here: I actually also got pretty excited reading this. The challenge is selling analyses with unknown outcomes to the bean counters who end up deciding. Blog posts like this help.
- tuyguntn 4y ago
  this is a problem on a company/culture level, a root cause is probably capitalism and pursuing for never ending growth. Because growth is most of the time translated as acquiring more users/customers. Eventually making more money out of them. Somewhere in the distant future, making money becomes cost saving, this is when companies start thinking about how can we cut the cost and such projects/explorations become relevant. Even then it is not easy to sell as a project. Because there are so many unknowns, discussion goes like: me: hey, I see spikes in CPU/mem usage, want to investigate why. Product: is it important now? me: I don't exactly know, but it might end up saving our instance costs. Product: how many instances we are talking about? me: I don't know yet how big is the problem, depends on what would be the result Product: how long does it take you to do a research? me: I don't know what is the problem yet, so I can't evaluate how long it takes to find it. Product: maybe we shouldn't do it then? me: (again) but it might end up saving our instance costs. Product: ok, do you think you can finish investigation in 2 days and share with me timeline and metrics this project would improve? me: ok, let's forget about this shit we made. what is a next "metric improving, highly visible" project we have? Product: cool, I want you to work on this super exciting "Move button from Page A to Page B" project
  - harha 4y ago
    Don’t think this is related to capitalism though, more about attitude to risk. If you always do the obvious things, you’ll end up as a mediocre company that might not survive. If you say yes to any project that comes to mind, you’ll end up burning all your money and end up with endless projects. It’s important to strike a balance and try to find the right investments others don’t see to create a competitive advantage. It helps if the culture allows for that. The questions still are relevant, you just don’t always need precise answers, Probe 100 instances and see if it surfaces more than once. Set up a list of things you think you’ll need to be doing and an initial proposition what you think might change. It’s the PMs job to then convince others that it’s worth taking the risk.
SergeAx 4y ago
Things like this mostly make sense on a Netflix' scale. One grade lower, and spinning up new VM is inherently cheaper than spending SWEs time on solving this low level problem.
blackoil 4y ago
Would any of the company really benefitted from this kind of analysis. NFLX serves Petabytes of data for a particular use case, majority of the Tier 1 companies would not have 100Mbps of traffic for their top service. Maybe there are 5-15 companies globally which will get value from this and they all will support for this. I have seen far more companies wasting resources on the projects of no value because the stakeholders believe themselves to be of same level as Netflix, Google.
- anonymoushn 4y ago
  It seems quite common that small or medium companies adopt highly complex architectures that could be avoided by caring slightly about performance
  - robertlagrant 4y ago
    Those (slightly complex) architectures and tools buy them availability and reproducibility as well as scalability.
    - anonymoushn 4y ago
      Typically negative availability and zero reproducibility?
trollied 4y ago
I used to be a performance engineer for telecoms systems. Tuning & fixing performance regressions saved millions in hardware costs & software licenses (think databases, licensed by core) in the late 90s/early 2000s. It was great fun, and highly rewarding (mentally). I'd love to get back into it.
jjav 4y ago
> These posts both make me violently aroused as well as profoundly depressed basically every employer I’ve ever had in 30 years doesn’t value such analysis. Couldn't agree more, except on the timeline. In my experience, the 90s and most of the 00s proper engineering like this was the norm in silicon valley and I loved it. It's the last 10-15 years that PMs have taken over and now it's all so superficial and uninteresting. I've moved away from hands-on development work because nobody is doing anything intellectually rewarding anymore. But I wish I was doing hard-core engineering like it used to be.
- sickill 4y ago
  What did you move to instead?
albertzeyer 4y ago
Well, depending on what you do, if performance or throughput matters, and usually it does, a 3.5x improvement in throughput means to save lots of money, so I'm sure most managers should see an advantage in that. In this case, the engineers had a clear expectation and good reasons that something was wrong and such throughput should be possible. The arguments for that are simple enough that every manager should also understand this. The other aspect is also latency but also here I would assume that managers should at least care a bit. They also have the old baseline to compare to, and see that it is suddenly much worse, so they should care about why that is and how to fix it.
throwaway0asd 4y ago
Frontend developers do not know how to test for performance and do not value performance at all. It’s both equally frustrating and baffling. So much so that I have this growing voice in my head telling me to quit my job and found a performance based consultancy. My observations contributing to the frustration: * Developers believe performance is speed. It isn’t. Performance is a difference in measure between two (or more) competing qualities, the performance gap measured in either duration or frequency. * Developers will double down on the prior mistake by making anecdotal observations without any forms of measurement and will vehemently argue their position in the form of a logic based value proposition. Example: X must be fast because Y and Z appear fast. * Developers will further hold their unmeasured position by isolating their observations from how things actually work. Software is a complex system of many instructions. Except for the most trivial of toy scripts nothing executes in isolation. For example developers follow the value proposition that a grain of sand is tiny and weighs little, so therefore does not contribute to the weight or size of the beach. An example is that developers will argue to the death using querySelectors to avoid walking the DOM but in Firefox DOM walking can be as fast as 250000x faster than querySelectorAll, which is indeed significant. I suspect there exist a variety motives for why developers reason things in this way, but from the outside it looks like a house of cards based upon argument from ignorance fallacies outputting really slow software by really defensive people.
- Someone 4y ago
  > Frontend developers […] do not value performance at all. I do not think that’s restricted to front-end developers. Back-end developers rarely worry about inserting yet another http request in code called from user action in a web browser, for example. A tenth of a second here, a tenth there, it all adds up. Product managers and users also don’t seem to care much or do not know how fast modern hardware is. I’ve frequently seen web page refreshes take 5 seconds or more without getting any user complaint, even when I explicitly ask them, and tell them how that could easily be halved.
- tbran 4y ago
  Can someone clarify this? > * Developers believe performance is speed. It isn’t. Performance is a difference in measure between two (or more) competing qualities, the performance gap measured in either duration or frequency. Maybe I'm a dummy, but if someone says "make a fast webapp", I'll be doing things like reducing requests, optimizing queries, making things smaller, using the right data structures, manipulating data in a fast and/or memory efficient way. This should result in lower loads, more stability, better availability, etc, so also "more performance". I do think of it as lots of focus on speed since, for the most part, I'm measuring how long it takes for a function/query/etc to run. Happy to check out some articles/keyword suggestions.
highwaylights 4y ago
It’s a self-solving problem over long enough timescales. There’s a reason YCombinator (as an example) insists on at least one strongly technical founder. Bean counters rot companies by not seeing the wood for the trees. The same applies to MBAs (no offence to anyone that has one). Engineering companies need to either be led by people the engineers respect, which means an engineer - or at the very least someone who understands the engineering thoroughly enough to get out of the way.
Cthulhu_ 4y ago
In my limited experience, there's not many companies that actually have the load that requires this level of fine-tuning. Or they don't actually own the software involved; we've had performance issues at the company I work for at the moment (as a consultant / contractor), caused by SAP and a product called Tibco, a visual editor to query and assemble data. Lack of transparency there. Anyway my point with that is that there are a lot more high level things that needed to be fixed - and this is the case in most, if not all organizations I've worked at - before you get to the lower levels of performance. And this probably applies everywhere. In this example, they have their software so optimized and tweaked already that it ended up being a low level JVM issue. Mind you / on the other hand, they mentioned something about subclasses and the like, it might be that a change in their Java code would have solved or amortized this issue as well.
- fnordpiglet 4y ago
  I’ll betcha any fleet with more than 100 machines in it hasn’t been tuned at a low level and could be 1-10 machines if so. That’s probably worth it. But you can’t convince an accountant that you can make money by spending it and at a certain size the accountants make the decisions.
gizmo 4y ago
It's a giant tragedy of the commons. For every individual company it makes sense to cut corners to get to market quicker or to save on engineering costs. For the industry as a whole it's a disaster.

skellington 4y ago

Yup, very few workplaces would know how to reward people with this skill level.

fnordpiglet 4y ago
Almost every engineer I’ve worked with have this skill level, but they’re not given the leeway, direction, or advice to get to the conclusion. Netflix must have an amazing culture in these areas.
- ilyt 4y ago
  Really that just need manager that presents the results to higher-ups as raw cash savings. "My team makes your team apps and make them run cheaper"
- bitcharmer 4y ago
  I've been a high performance systems engineer for a number of investment banks over the last 10+ years and people with deep knowledge like that are quite rare; that's why we get paid handsomely. Mind sharing what field you work in that gathers so much talent like that?
  - fnordpiglet 4y ago
    Knowledge != skill. With the skill you can acquire the knowledge given the right conditions and motivation. I would say the vast majority of engineers I work with have never had the need or opportunity to invest the time or experiential opportunities to acquire the knowledge required - but they COULD. I actually find that quite sad, and hence my depression I noted. Nowadays this is extending all the way through school. Machine architecture and low level machine microcode etc isn’t universally taught in CS programs any more. I’ve worked in all the industries, including a long time in yours. We probably even overlapped or maybe worked together!
saagarjha 4y ago
You have to know how to justify your work. Places like Netflix have teams like these because they literally print money, with each engineer often saving the company millions if not tens of millions of dollars annually. It’s hard to argue with results.
lmm 4y ago
Very few workplaces value this skill because very few workplaces have a use for it. One service now runs 3x faster? You reduced the AWS bill for this one service by 2/3? That's not going to change the bottom line enough for anyone to care about, unless you're, well, Netflix.

abrookewood 4y ago

I think you need to work at companies that are large enough to allow for people to dedicate their time to this level of analysis. Smaller companies are too busy to have staff focussing on things like this.

xxpor 4y ago
Seems like poor analysis on their part then. If they're leaving 3.5x perf on the floor, they're not spending their money very well.
- loeg 4y ago
  A lot of businesses reasonably use Python and Ruby, for example.
  - snovv_crash 4y ago
    If you can run 1 server instead of 3 though... that means no ansible, no kubernetes, no wild deployment strategies, 1/3 the likelihood of hardware failure, etc
    - maccard 4y ago
      This isn't true, at all. We are not "high availability" by any means but running on one server has significant risks, _especially_ if you don't use some form of automated provisioning. A destructive app logic bug is far more likely than a hardware failure, and in both cases the impact on your service is significant if you have one instance, but likely manageable if you have more than one.
    - heurisko 4y ago
      Why wouldn't you use ansible to manage one server? It's a great tool with a low learning curve that just requires SSH.
- magicalhippo 4y ago
  Leaving 3.5x perf of what on the floor though? We're working on applications that's either waiting for the disk, waiting for the db, waiting for some http server or waiting for the user. None of our customers will notice the difference if my button click event handler takes 50ms instead of 10ms, or if the integration service that processes a few dozen files per day spends 5 seconds instead of 1 second. I'll easily trade 5x performance in 99% of my code if it makes me produce 5x more features, because most of the time my code runs in just a few milliseconds at a time anyway. Of course, I'm weary of big-Oh, that one will bit ya. But a mere 3.5x is almost never worth chasing for us.

newZWhoDis 4y ago

Fuck “project managers”. The second you are subjected to one as an engineer you are being told you’re in the wrong place. I don’t care what the stock options are, I don’t care what the free morning deviled eggs are like, that place is cancer and you need to escape it.

harha 4y ago

PM here: I actually also got pretty excited reading this. The challenge is selling analyses with unknown outcomes to the bean counters who end up deciding. Blog posts like this help.

tuyguntn 4y ago
this is a problem on a company/culture level, a root cause is probably capitalism and pursuing for never ending growth. Because growth is most of the time translated as acquiring more users/customers. Eventually making more money out of them. Somewhere in the distant future, making money becomes cost saving, this is when companies start thinking about how can we cut the cost and such projects/explorations become relevant. Even then it is not easy to sell as a project. Because there are so many unknowns, discussion goes like: me: hey, I see spikes in CPU/mem usage, want to investigate why. Product: is it important now? me: I don't exactly know, but it might end up saving our instance costs. Product: how many instances we are talking about? me: I don't know yet how big is the problem, depends on what would be the result Product: how long does it take you to do a research? me: I don't know what is the problem yet, so I can't evaluate how long it takes to find it. Product: maybe we shouldn't do it then? me: (again) but it might end up saving our instance costs. Product: ok, do you think you can finish investigation in 2 days and share with me timeline and metrics this project would improve? me: ok, let's forget about this shit we made. what is a next "metric improving, highly visible" project we have? Product: cool, I want you to work on this super exciting "Move button from Page A to Page B" project
- harha 4y ago
  Don’t think this is related to capitalism though, more about attitude to risk. If you always do the obvious things, you’ll end up as a mediocre company that might not survive. If you say yes to any project that comes to mind, you’ll end up burning all your money and end up with endless projects. It’s important to strike a balance and try to find the right investments others don’t see to create a competitive advantage. It helps if the culture allows for that. The questions still are relevant, you just don’t always need precise answers, Probe 100 instances and see if it surfaces more than once. Set up a list of things you think you’ll need to be doing and an initial proposition what you think might change. It’s the PMs job to then convince others that it’s worth taking the risk.

SergeAx 4y ago

Things like this mostly make sense on a Netflix' scale. One grade lower, and spinning up new VM is inherently cheaper than spending SWEs time on solving this low level problem.

blackoil 4y ago

Would any of the company really benefitted from this kind of analysis. NFLX serves Petabytes of data for a particular use case, majority of the Tier 1 companies would not have 100Mbps of traffic for their top service. Maybe there are 5-15 companies globally which will get value from this and they all will support for this. I have seen far more companies wasting resources on the projects of no value because the stakeholders believe themselves to be of same level as Netflix, Google.

anonymoushn 4y ago
It seems quite common that small or medium companies adopt highly complex architectures that could be avoided by caring slightly about performance
- robertlagrant 4y ago
  Those (slightly complex) architectures and tools buy them availability and reproducibility as well as scalability.
  - anonymoushn 4y ago
    Typically negative availability and zero reproducibility?

trollied 4y ago

I used to be a performance engineer for telecoms systems. Tuning & fixing performance regressions saved millions in hardware costs & software licenses (think databases, licensed by core) in the late 90s/early 2000s. It was great fun, and highly rewarding (mentally). I'd love to get back into it.

jjav 4y ago

> These posts both make me violently aroused as well as profoundly depressed basically every employer I’ve ever had in 30 years doesn’t value such analysis. Couldn't agree more, except on the timeline. In my experience, the 90s and most of the 00s proper engineering like this was the norm in silicon valley and I loved it. It's the last 10-15 years that PMs have taken over and now it's all so superficial and uninteresting. I've moved away from hands-on development work because nobody is doing anything intellectually rewarding anymore. But I wish I was doing hard-core engineering like it used to be.

sickill 4y ago
What did you move to instead?

albertzeyer 4y ago

Well, depending on what you do, if performance or throughput matters, and usually it does, a 3.5x improvement in throughput means to save lots of money, so I'm sure most managers should see an advantage in that. In this case, the engineers had a clear expectation and good reasons that something was wrong and such throughput should be possible. The arguments for that are simple enough that every manager should also understand this. The other aspect is also latency but also here I would assume that managers should at least care a bit. They also have the old baseline to compare to, and see that it is suddenly much worse, so they should care about why that is and how to fix it.

throwaway0asd 4y ago

Frontend developers do not know how to test for performance and do not value performance at all. It’s both equally frustrating and baffling. So much so that I have this growing voice in my head telling me to quit my job and found a performance based consultancy. My observations contributing to the frustration: * Developers believe performance is speed. It isn’t. Performance is a difference in measure between two (or more) competing qualities, the performance gap measured in either duration or frequency. * Developers will double down on the prior mistake by making anecdotal observations without any forms of measurement and will vehemently argue their position in the form of a logic based value proposition. Example: X must be fast because Y and Z appear fast. * Developers will further hold their unmeasured position by isolating their observations from how things actually work. Software is a complex system of many instructions. Except for the most trivial of toy scripts nothing executes in isolation. For example developers follow the value proposition that a grain of sand is tiny and weighs little, so therefore does not contribute to the weight or size of the beach. An example is that developers will argue to the death using querySelectors to avoid walking the DOM but in Firefox DOM walking can be as fast as 250000x faster than querySelectorAll, which is indeed significant. I suspect there exist a variety motives for why developers reason things in this way, but from the outside it looks like a house of cards based upon argument from ignorance fallacies outputting really slow software by really defensive people.

Someone 4y ago
> Frontend developers […] do not value performance at all. I do not think that’s restricted to front-end developers. Back-end developers rarely worry about inserting yet another http request in code called from user action in a web browser, for example. A tenth of a second here, a tenth there, it all adds up. Product managers and users also don’t seem to care much or do not know how fast modern hardware is. I’ve frequently seen web page refreshes take 5 seconds or more without getting any user complaint, even when I explicitly ask them, and tell them how that could easily be halved.
tbran 4y ago
Can someone clarify this? > * Developers believe performance is speed. It isn’t. Performance is a difference in measure between two (or more) competing qualities, the performance gap measured in either duration or frequency. Maybe I'm a dummy, but if someone says "make a fast webapp", I'll be doing things like reducing requests, optimizing queries, making things smaller, using the right data structures, manipulating data in a fast and/or memory efficient way. This should result in lower loads, more stability, better availability, etc, so also "more performance". I do think of it as lots of focus on speed since, for the most part, I'm measuring how long it takes for a function/query/etc to run. Happy to check out some articles/keyword suggestions.

highwaylights 4y ago

It’s a self-solving problem over long enough timescales. There’s a reason YCombinator (as an example) insists on at least one strongly technical founder. Bean counters rot companies by not seeing the wood for the trees. The same applies to MBAs (no offence to anyone that has one). Engineering companies need to either be led by people the engineers respect, which means an engineer - or at the very least someone who understands the engineering thoroughly enough to get out of the way.

Cthulhu_ 4y ago

In my limited experience, there's not many companies that actually have the load that requires this level of fine-tuning. Or they don't actually own the software involved; we've had performance issues at the company I work for at the moment (as a consultant / contractor), caused by SAP and a product called Tibco, a visual editor to query and assemble data. Lack of transparency there. Anyway my point with that is that there are a lot more high level things that needed to be fixed - and this is the case in most, if not all organizations I've worked at - before you get to the lower levels of performance. And this probably applies everywhere. In this example, they have their software so optimized and tweaked already that it ended up being a low level JVM issue. Mind you / on the other hand, they mentioned something about subclasses and the like, it might be that a change in their Java code would have solved or amortized this issue as well.

fnordpiglet 4y ago
I’ll betcha any fleet with more than 100 machines in it hasn’t been tuned at a low level and could be 1-10 machines if so. That’s probably worth it. But you can’t convince an accountant that you can make money by spending it and at a certain size the accountants make the decisions.

gizmo 4y ago

It's a giant tragedy of the commons. For every individual company it makes sense to cut corners to get to market quicker or to save on engineering costs. For the industry as a whole it's a disaster.