4 ms·

And yet, the large organizations with R&D firms—making the most exciting greenfield software—almost exclusively focus their engineering efforts on squeezing the

And yet, the large organizations with R&D firms—making the most exciting greenfield software—almost exclusively focus their engineering efforts on squeezing the most software out of the lowest amount of hardware. They care deeply about every CPU cycle. You could argue that the fallacy of premature optimization has fallen many a line of business app. But if you're an engineer and you aspire for greatness, microoptimize. That's the rarer, harder-earned, harder-to-learn skill.

dang 10y ago
I always think of this comment which stuck in my mind years ago: "Consistently the hard technical problems are performance problems." https://news.ycombinator.com/item?id=1835859 https://news.ycombinator.com/item?id=1835859
- quizotic 10y ago
  And I've heard it said that lowering latency is harder than increasing throughput ... though whether it's more important probably depends on the context.
SomeCallMeTim 10y ago
I see this for * Machine learning * Computer graphics * Financial ...but not much else. The Facebook app is a nightmare of terrible design that has something like 100Mb of generated code to do something that should take a couple megabytes at most. And they are a large organization with R&D by any definition.
- doctorpangloss 10y ago
  To your particular point, Facebook maintains multiple versions of its client, including a "basic mobile" version (https://mbasic.facebook.com https://mbasic.facebook.com) and a "touch" version (https://touch.facebook.com https://touch.facebook.com), among other idiosyncrasies between its various platforms. They put in a lot of engineering effort to make the best client for the appropriate device. With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs. But in terms of R&D, you could argue that Oculus is their major public R&D project. That project requires lots of performance optimization. Indeed, that is their greatest obstacle. Maybe you could call that computer graphics, but that's missing the point. Making the sensors and signal processors work on battery-powered hardware (the end goal) is really challenging. And then consider all the backend services that will serve these peripherals. This Premature Optimization Koan is really just a side effect of enumerative thinking. Like if you think about the problem hard enough, you'll be able to enumerate all of its aspects and select an appropriate solution. That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them. In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems. In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere. I'm sure at Facebook they share your concern. They probably work extremely hard to reduce the huge amount of code their iOS client is built from. I'm advocating that people train to improve performance, not to ignore the problem or dismiss it as "premature optimization," at least if they aspire to be engineers.
  - SomeCallMeTim 10y ago
    I do game for a living. I guess I assumed that people knew that games were optimized, though they usually don't fall under "enterprise." Oculus falls under games for me, which is unusual to find in an enterprise company, and frankly probably isn't being run much like the rest of Facebook anyway, so the point is rather moot. >That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them. I'm a big fan of optimization, and yet I've done exactly the latter and come to the conclusion that a particular admin backend that would never have more than, say, a couple dozen users EVER would be fine to be written in Ruby. It was an internal tool, and the available engineers were efficient in Rails, so I gave a green light for them to write the admin tools in Rails. There wasn't even a debate. I just said yes, go. But when you're writing a service that should have a solid front-end that can scale, you need to design scalability into the architecture. It seems to rarely happen. >With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs. Cassandra is good and well optimized. Point. >In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems. Those are the things that came to mind. No, I'm sure I didn't hit all possible high-performance issues. But that's probably the 80th percentile. And at least 95% of "enterprise" problems are not treated with the respect they should be toward optimizing. That I have from several personal sources. >They put in a lot of engineering effort to make the best client for the appropriate device. They put in 100x as much engineering effort as they should have needed to to create a solid mobile app, and the result is a monstrosity of garbage. Several indie developers have produced replacement Facebook apps that are far better, faster, and would actually be more robust if Facebook weren't actively trying to prevent them from working. So you can't tell me you need a hundred developers writing generated code to create two apps (the main Facebook app and Facebook Messenger; if you want to read messages, the main app has that DISABLED with a nag screen to download Messenger if you want to read messages) are actually even remotely competent (as a group) if some developer in their bedroom can crank out a better app in their spare time. I've also written an entire app from the ground up that had about 80% of the Facebook app's functionality, written against some of the worst backend APIs you can imagine (a mix of non-standardized, inconsistent XMLRPC, JSON, and other APIs) and it took me, writing code entirely by myself, about 5 weeks. To a finished, polished product. And that's at only about 30 hours a week. >In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere. I don't dispute that they should pay attention to performance. I just dispute that enterprise developers (excluding exceptional situations like Cassandra) actually optimize architecturally. Sometimes they can't for structural reasons, since ideal optimization needs to happen across teams and domains, which can be difficult in an environment where each team controls a domain.
rubber_duck 10y ago
1. Modern CPUs have "mini OS-es in them", difference between two generations of CPUs from the same manufacturer, even two models in the same genration, can mean that approach A is faster than approach B, let alone cross platform and cross architecture optimization. 2. Stuff you're running is usually not running in a vacuum but as a part of a bigger system, up front micro optimizing without measuring is again blind guessing which ends up being wrong and a waste of time. Micro optimization is a relic of the past, something that embedded/console/driver developers working on specific hardware need to care about - for the rest this stuff it's just a pointless guessing game - I've seen a few old C developers stuck in their ways try to do this to look smart and when you actually go to measure it ended up being wrong because their assumptions are outdated. Spending your time on high level system optimization seems much more worth while (picking the right algorithms, exposing the right kind of interfaces for efficient consumption, etc.).
- doctorpangloss 10y ago
  > embedded/console/driver developers working on specific hardware need to care about In other words, things that great engineers need to care about :) I'm not an embedded developer, but to reiterate, I'm just advocating for a focus on performance if you aspire to be a great engineer. This is different than being a great businessperson, or a great software developer kit / library designer.
  - rubber_duck 10y ago
    >In other words, things that great engineers need to care about :) Really ? I feel like most cutting edge engineering is done in building distributed systems, ML, high performance computing, etc. Micro optimization is where it was last century.
    - quickben 10y ago
      That is just what gets most advertisement time these days. It sells well as a news item. AI, ML, self driving cars. But, when you have to ship 50k IOT devices and they have to be under $20. Your bosses will squeeze you for every ounce of optimization.
justicezyx 10y ago
Your statement is absolutely wrong, at least for Google. Google, above anything, put readability and architecture soundness first. And that pays off, as fixing micro performance issue in a sound architecture is always cheap, the reverse, however is expensive. (Fixing problematic architecture while maintaining good performance)

dang 10y ago

I always think of this comment which stuck in my mind years ago: "Consistently the hard technical problems are performance problems." https://news.ycombinator.com/item?id=1835859 https://news.ycombinator.com/item?id=1835859

quizotic 10y ago
And I've heard it said that lowering latency is harder than increasing throughput ... though whether it's more important probably depends on the context.

SomeCallMeTim 10y ago

I see this for * Machine learning * Computer graphics * Financial ...but not much else. The Facebook app is a nightmare of terrible design that has something like 100Mb of generated code to do something that should take a couple megabytes at most. And they are a large organization with R&D by any definition.

doctorpangloss 10y ago
To your particular point, Facebook maintains multiple versions of its client, including a "basic mobile" version (https://mbasic.facebook.com https://mbasic.facebook.com) and a "touch" version (https://touch.facebook.com https://touch.facebook.com), among other idiosyncrasies between its various platforms. They put in a lot of engineering effort to make the best client for the appropriate device. With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs. But in terms of R&D, you could argue that Oculus is their major public R&D project. That project requires lots of performance optimization. Indeed, that is their greatest obstacle. Maybe you could call that computer graphics, but that's missing the point. Making the sensors and signal processors work on battery-powered hardware (the end goal) is really challenging. And then consider all the backend services that will serve these peripherals. This Premature Optimization Koan is really just a side effect of enumerative thinking. Like if you think about the problem hard enough, you'll be able to enumerate all of its aspects and select an appropriate solution. That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them. In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems. In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere. I'm sure at Facebook they share your concern. They probably work extremely hard to reduce the huge amount of code their iOS client is built from. I'm advocating that people train to improve performance, not to ignore the problem or dismiss it as "premature optimization," at least if they aspire to be engineers.
- SomeCallMeTim 10y ago
  I do game for a living. I guess I assumed that people knew that games were optimized, though they usually don't fall under "enterprise." Oculus falls under games for me, which is unusual to find in an enterprise company, and frankly probably isn't being run much like the rest of Facebook anyway, so the point is rather moot. >That could mean enumerating all the possible applications of high performance software, or it could mean enumerating all of the strategic goals of a piece of business software and identifying that performance isn't one of them. I'm a big fan of optimization, and yet I've done exactly the latter and come to the conclusion that a particular admin backend that would never have more than, say, a couple dozen users EVER would be fine to be written in Ruby. It was an internal tool, and the available engineers were efficient in Rails, so I gave a green light for them to write the admin tools in Rails. There wasn't even a debate. I just said yes, go. But when you're writing a service that should have a solid front-end that can scale, you need to design scalability into the architecture. It seems to rarely happen. >With respect to the backend of their main application, they created things like Cassandra to meet their unique database needs. Cassandra is good and well optimized. Point. >In the former case, you're going to feel like you've enumerated enough (e.g., three examples) and then come short of the actual space of high-performance problems. Those are the things that came to mind. No, I'm sure I didn't hit all possible high-performance issues. But that's probably the 80th percentile. And at least 95% of "enterprise" problems are not treated with the respect they should be toward optimizing. That I have from several personal sources. >They put in a lot of engineering effort to make the best client for the appropriate device. They put in 100x as much engineering effort as they should have needed to to create a solid mobile app, and the result is a monstrosity of garbage. Several indie developers have produced replacement Facebook apps that are far better, faster, and would actually be more robust if Facebook weren't actively trying to prevent them from working. So you can't tell me you need a hundred developers writing generated code to create two apps (the main Facebook app and Facebook Messenger; if you want to read messages, the main app has that DISABLED with a nag screen to download Messenger if you want to read messages) are actually even remotely competent (as a group) if some developer in their bedroom can crank out a better app in their spare time. I've also written an entire app from the ground up that had about 80% of the Facebook app's functionality, written against some of the worst backend APIs you can imagine (a mix of non-standardized, inconsistent XMLRPC, JSON, and other APIs) and it took me, writing code entirely by myself, about 5 weeks. To a finished, polished product. And that's at only about 30 hours a week. >In the latter case, in my personal experience developing a line of business application, it turns out that performance eventually starts to matter. It sort of becomes the hardest problem, like people have said elsewhere. I don't dispute that they should pay attention to performance. I just dispute that enterprise developers (excluding exceptional situations like Cassandra) actually optimize architecturally. Sometimes they can't for structural reasons, since ideal optimization needs to happen across teams and domains, which can be difficult in an environment where each team controls a domain.

rubber_duck 10y ago

1. Modern CPUs have "mini OS-es in them", difference between two generations of CPUs from the same manufacturer, even two models in the same genration, can mean that approach A is faster than approach B, let alone cross platform and cross architecture optimization. 2. Stuff you're running is usually not running in a vacuum but as a part of a bigger system, up front micro optimizing without measuring is again blind guessing which ends up being wrong and a waste of time. Micro optimization is a relic of the past, something that embedded/console/driver developers working on specific hardware need to care about - for the rest this stuff it's just a pointless guessing game - I've seen a few old C developers stuck in their ways try to do this to look smart and when you actually go to measure it ended up being wrong because their assumptions are outdated. Spending your time on high level system optimization seems much more worth while (picking the right algorithms, exposing the right kind of interfaces for efficient consumption, etc.).

doctorpangloss 10y ago
> embedded/console/driver developers working on specific hardware need to care about In other words, things that great engineers need to care about :) I'm not an embedded developer, but to reiterate, I'm just advocating for a focus on performance if you aspire to be a great engineer. This is different than being a great businessperson, or a great software developer kit / library designer.
- rubber_duck 10y ago
  >In other words, things that great engineers need to care about :) Really ? I feel like most cutting edge engineering is done in building distributed systems, ML, high performance computing, etc. Micro optimization is where it was last century.
  - quickben 10y ago
    That is just what gets most advertisement time these days. It sells well as a news item. AI, ML, self driving cars. But, when you have to ship 50k IOT devices and they have to be under $20. Your bosses will squeeze you for every ounce of optimization.

justicezyx 10y ago

Your statement is absolutely wrong, at least for Google. Google, above anything, put readability and architecture soundness first. And that pays off, as fixing micro performance issue in a sound architecture is always cheap, the reverse, however is expensive. (Fixing problematic architecture while maintaining good performance)