12 ms·

Performance excuses debunked

141 points by brhsagain 3y ago

deleted 3y ago
[deleted]
leoncaet 3y ago
Casey does it again.
naught00 3y ago
What is funny is many of the exact excuses were made in this thread: https://news.ycombinator.com/item?id=34966137 https://news.ycombinator.com/item?id=34966137
- AbsoluteCabbage 3y ago
  yeah, and now people here are making the new excuses we already predicted lol
db48x 3y ago
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 MBs off the total machine code size and bought us enough runway to keep development going. Wow. You know you have made bad choices when you need a desperation play like that.
typon 3y ago
I kind of wish Casey would stop making these videos. I'm being completely serious when I say that this kind of knowledge and mindset gives you pretty much a superpower in the industry. Everywhere I go people think I am amazing, even though I just know a few things about performance and low level programming (like, enough to be amazing for "mere mortals", but someone at RAD would snicker at my knowledge). If the industry doesn't care, it's simply a competitive advantage for you as an individual.
- db48x 3y ago
  I know what you mean; I’ve made a lot of money by cleaning up other people’s messes. Still, I’m pretty sure I’d prefer a world where software was just better overall and there were fewer messes.
  - jiggawatts 3y ago
    I had a two year stint where I ran around doing "database optimisation" projects that were a week of billable time at consulting rates, and my entire "toolkit" was to simply flip three boolean flags from bad to good values. Now in the public cloud my gimmick is to switch VMs over to current-generation AMD EPYC models, and then collect my pay-check for the "performance tuning". The fruit is hanging so low that I'm trampling over it. There's no actual fruit picking going on.
    - runlaszlorun 3y ago
      I’ve always enjoyed perf testing and optimization but figured it’d be tough to reliably find clients staying in that niche. Any tips?
      - jiggawatts 3y ago
        I never got enough of this type myself, they were just random gigs between other types of project work. IMHO, the IT industry as a whole has internalised wastefulness and accepted poor performance because there's an expectation that "CPU upgrades will fix it". Over five decades that was true, it's just that few organisations have adapted to the new reality of only ~20% uplifts in single-thread performance over the last decade or so. In principle, there ought to be many more optimisation projects, but unfortunately they're still rare.
    - db48x 3y ago
      My favorite is to raise the Postgres work_mem value until their queries stop using disk buffers.
- SPBS 3y ago
tmtvl 3y ago
I like the positive message he leaves us with: the barrier to entry for good performance is as low as it's ever been. You can get good performance without needing hand-coded assembly SIMD vector optimizations.
- kragen 3y ago
  also tho the barrier to entry for hand-coded assembly simd vector optimizations is as low as it's ever been
paul_grisham 3y ago
Obviously performance matters, it's just a question of opportunity cost. The article just takes some absolute statements about performance not mattering and argues against them, which is obvious and pointless. They ignore that the benefit/cost ratio for Facebook's performance improvements is vastly different to that of the average software company with tens of employees and some SaaS CRUD app. Their own example of Uber's app rewrite due to their previous architecture not being good enough is an example against their point - they got to where they were with imperfect architecture, then later fixed it once the opportunity cost tradeoff made sense.
- db48x 3y ago
  Did you read the same article I did? He explicitly states exactly that: > For example, one argument would be that the evidence I’ve presented here is consistent with a strategy of quickly shipping “version one” with poor performance, then starting work on a high-performance “version two” to replace it. That would be completely consistent with the evidence we see. > But even if that turns out to be true, it still means programmers have to care about performance! It just means they need to learn two modes of programming: “throw-away”, and “performant”. There would still be no excuse for dismissing performance as a critical skill, because you always know the throw-away version has to be replaced with a more performant version in short order. > That kind of argument is great. We should have it. What we should not have are excuses — claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle, so developers simply don’t have to learn about it.
  - paul_grisham 3y ago
    > But even if that turns out to be true, it still means programmers have to care about performance! Who is he even talking to at this point? He's arguing that performance has more than 0% relevance to software, which no-one on disagrees with. >claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle No-one says performance never matters. People disagree on what performance merits optimization. A 5s to 1s page load improvement is massive, a .5s to .1s improvement starts hitting diminishing returns in user experience.
alphanumeric0 3y ago
"With so much evidence refuting the five excuses, hopefully it is clear that they are ridiculous. They are completely invalid reasons for the average developer, in the common case, to dismiss concerns about software performance. They should not be taken seriously in a professional software development context." I believe the average developer should care somewhat about performance, and depending on their industry they might need to care a lot, but I'm not so convinced for the average case. The average developer is not working on FAANG-sized codebases. Also, I'd imagine any large systems built up over the years that are refactored would likely see great performance gains. That's just the nature of long-term software.
magicalhippo 3y ago
There's often a lot of low hanging performance fruits when writing code. Need to loop over some order lines and find distinct article numbers? Use a hash-based set with O(1) access, not just a list which will have O(n). If not you'll end up writing an O(n^2) routine for no good reason, which will work swimmingly on you 10 line test order and cause grief in production. I don't think a lot about performance most of the time, just enough to try to avoid silly stuff.
- kuvlopa 3y ago
  Even better; let the database handle it if you have one, calling tons of extra data down the pipe is very slow compared to even slow iteration, and even the slow iteration is often fast if you only pull relevant data
- citrin_ru 3y ago
  O(1) hash vs O(N) array loop is a bit more complicated - for small N linear search can be faster - no need to compute a hash function, if data stored directly in the array (instead of pointers) then there will be less CPU cache misses. Exact threshold can be determined in a benchmark but the key here here is that N should be bounded - if a user can manipulate N to be huge and you have O(N^2) somewhere in your code it is not only performance but also a potential security problem - a DoS vector. IMHO the key to not writing silly code is to have a good mental model of how your code works and how it fits into a bigger picture which includes OS, hardware, network and other services (like a DB). This model should be updated from time to time based on benchmarks and metrics from production services but once you have this knowledge it takes almost no additional time to apply this in practice and even if you'll be taking shortcuts to save development time you'll be better aware of this tradeoff.
  - magicalhippo 3y ago
    Yeah I was thinking in a CRUD setting, where I live, where you won't be doing the operation too often. In that case the inefficiency for small N is a non-issue vs the pain of a O(N^2).
  - never_inline 3y ago
globalreset 3y ago
Scale / data set size tends to expose performance issues. But that's a good problem to have. Yes, it would be great if all developers could write performant code, but let's face it - there's only so many hours in a day and days in a week. Developers already struggle to keep up with all the required knowledge. It's not that people don't want to be competent. We're building more and more complex things while expanding the number of people employed building software which means average skill level is probably slightly decreasing.
- HideousKojima 3y ago
  >It's not that people don't want to be competent. There are plenty of devs that are just there for the paycheck and have minimal passion or interest in learning more.
  - hikawaii 3y ago
    It's often not about learning more, the techniques and methods in Casey Muratori's streams are often very well understood and even taught in school. Dismissing people as just being there for the paycheck - that's you and every other developer. The number of people who produce useful OSS projects because they just like the project and aren't filling out resume points to hope bigtech senpai notices them is astronomically small, and usually the sort of political project nobody wants to touch.
  - globalreset 3y ago
    True, but it's normal in many professions that lots of people working in it are not passionate and industry just deals with it. We can't expect that average software developer is going to be passionate and spending their weekends honing every-increasing set of skills: technologies, platforms, tools, etc.
    - eastbound 3y ago
      Can’t we? The entire startup ecosystem is based on extracting from the economy people who are motivated, do their work efficiently and learn high skills, and giving them millionaire-or-bankrupt stakes to ensure they’re committed.
charlie0 3y ago
Starts off with premise of why performance matters and why you should care. Proceeds to only top 10 biggest tech companies as examples. Are you working in a FAANG? If not, this article does nothing to convince me that I should care about performance (even though I kinda do).
- readthenotes1 3y ago
  It convinced me that there must be scads of failed attempts that focused on performance first.
- omnicognate 3y ago
  He's a games programmer. He's using examples outside his industry to illustrate that his points aren't specific to games (excuse 4). What industry would you like him to talk about?
  - kragen 3y ago
    smes doing hourly contract crud development startups
    - vgatherps 3y ago
      To be fair, performance is usually part of the “does it work in real life” question, it’s just that doing sensible things is probably good enough. The crud app at your startup probably doesn’t need to worry about cutting optimizing every last millisecond off of page load or scaling across some k8s cluster or whatever. But I’ve stopped using (or never seriously tried) otherwise good services because of flow-stopping performance problems i.e. multi-second page loads, stuttering pages, battery drain, etc.
      - kragen 3y ago
        i'm not the one you need to convince, i'm just saying what marketing collateral i need to convince the skeptical: specific, detailed stories from those sectors, ideally at least somewhat verifiable (not a pseudonymous comment your wife's cousin read on hn) but specific and detailed goes a long way
klodolph 3y ago
Also note that “Facebook reduces the storage it needs to store user photos by 50%” is something that contributes, like, nine-figure dollar savings to the bottom line. Facebook does all sorts of crazy stuff to improve performance–I don’t think many people who work there understand how the javascript bundler works, for example. It’s a lot more than something like WebPack.
phendrenad2 3y ago
> a situation that sometimes happens does not support the use of a statement as a blanket excuse I feel like pointing this out has become table stakes in any conversation anywhere. Thinking of printing it out on a card and carrying it around.
devjab 3y ago
Performance matters, but I’m not sure people are saying that it doesn’t. I’m not a performance “first” type of programmer by any means, but I always keep it in the back of my mind, what I also tend to keep in the back of my mind is cost. I recently “inherited” a couple of back-end services when a developer left our company. It turned out that the code was terrible and that they haven’t used, any, or our helper tools. Since we use Typescript everywhere ignoring our quite opinionated and a little fascist linter rules is almost impossible, but the developer in question had the authority to turn it off, which they did and in doing so shot themselves completely in the foot. The back-end services were developed in JavaScript more than Typescript and since both our linters and usual teat pipelines were disabled, and since it’s software that has been developed over almost a years worth of changes, it was just horrible. We’re talking loops comparing values, that were probably there once but are now just sorting things as undefined === undefined kinds of terrible. The performance was also atrocious. Basically what the service did was gather info on a couple of thousand projects and link them with tens of thousands documents in Sharepoint, but because it was build wrong, it wasn’t pulling the correct documents and it was taking 5-10 minutes each run time. It’s now running at around 10 seconds for its complete run time. Which is a massive performance improvement, and it’ll be even better once I finish building the cashing. So you might think I’m inclined to agree with the article, but I didn’t rewrite it because of its poor performance, I rewrote it because it didn’t work correctly and the performance gains were simply a happy “coincidence”. This is because the performance didn’t really matter. Yes, it was costing us at most $77 a over our 3 year Azure contract, but the time I spent rebuilding it cost the company almost exactly $1500. Those $1500 were well spent because it wasn’t working, but would they have been well spent in terms of performance? Not really. That being said, it wouldn’t take a lot of those services to become expensive, so it’s not like the author is really wrong either. It’s just that I’m confused with whom he is arguing.
- camgunz 3y ago
  The points he lays out at the beginning of his post are basically Martin Fowler's (and thus 90% of enterprise engineers') position in Refactoring. Fowler was probably responding to engineers making code inscrutable in the name of (dubious claims of) performance, and Muratori here is responding to the ensuing overreaction.
tomas789 3y ago
One more datapoint here. Good performance helps avoiding scaling problems. One example is an API I recently wrote. I was initially in Python and performance was OK I guess. I had to take care of horizontal scaling, how to manage the infrastructure, a bit of distributed scaling and all the other goodies that come along. By rewriting it in Rust and caring about performance I was able to run it on a single machine (which is ok for that product). That saved me quite some hustle. And my average response time went from 300ms to 29ms. If the app would scale a lot I would still need to go into all that trouble but at this early stage the benefits of really simple infrastructure is immense.
- musicale 3y ago
  Rewriting your C++ app in Python can help it scale from 1000 servers to 100,000 servers without an increase in total throughput (although latency may increase.) It can also scale your power usage up from 1KW to 1000KW. If you have a datacenter that is 99% idle, you may be able to maximize utilization by switching to Python.
ftxbro 3y ago
So the biggest tech companies in the world, now that they are huge, are making their programs faster. OK. To me, that says maybe the reason they beat their competition to becoming huge, was that they wrote a lot of features quickly and didn't nitpick about performance. Maybe the ones that wrote efficient high performance code from the start stayed small or were out-competed.
- simula67 3y ago
  When I was starting out using the web, one of the reasons for using Google over competition was that it was fast. The home page loaded fairly quickly, and the results were instantaneous. Google spent a lot of time optimizing for performance during its early years and they were explicit about it.
  - yonixw 3y ago
    The problem with this argument is that there are many contradicting sample points. Like chrome. On one hand, Google created the JS-Engine V8 which finally made the web fast. But on the other hand, did nothing to fix the RAM issues that came with that until recent years. So I see it as "Let's do performance only when it's an advantage for us". Which all of the OP examples falls to. OP want to conclude from that that performance=advantage so everyone need to work on it all the time (because who will ignore advantages?), but all we see is if(performance=advantage){refactor}. Which does not support his conclusion. Example cases are startups, monopoles, high cost (like refactoring bank code) etc. which have other silos of advantage.
- AbsoluteCabbage 3y ago
  you realize that you are literally parroting talking points that we have already predicted you would come up with as additional excuses, right?
  - ftxbro 3y ago
    I mean I'm not parroting anything, but OK I will also agree that this isn't an especially mind blowing point that I was making. Probably a lot of other people would make that same point too!
yen223 3y ago
It bothers me that all the examples provided shows that companies care about performance ("Company $X made their app 20% faster"), but none of them talks about the impact of caring about performance ("Company $X made their app 20% faster, and saw a 200% increase in revenue as a result")
- kjksf 3y ago
  Casey obviously can't know what financial impact of those changes were at Twitter, Facebook or Microsoft. But Ockham Razor tells us they did spend significant effort on performance because they had a reason to do it.
xupybd 3y ago
The author fails to make the business case for their arguments. Performance tuning costs money. Don't spend money with no expected ROI. Yes at Facebook with end user facing software it is crucial to get performance right. If you're running payroll at a 3 person company it doesn't matter if the software is inefficient. Most of the time it's Excel, and that is not the most efficient way to do those calculations. But it's not worth investing in a better solution until processing payroll is a problem.
School-Cotton 3y ago
Absolutely hilarious and ridiculous to claim that Facebook cares about iOS app performance. In reality they have a few dozen people who care making heroic technical fixes to mitigate the damage of thousands of people adding mountains of mediocre crap to the apps. If you care about performance from the beginning, you would never even get to the point where you’re saying “oh crap, our messenger app can barely run, let’s rewrite it in C”.
- Jach 3y ago
  Late to the party but this is the same point of disagreement I had with the video, though overall it was fine. Why do these performance efforts and rewrites at the companies he listed take years? How complex is the uber eats app anyway? Well, it's because most engineers at those (and other) companies do not care or know about performance, and when it actually matters to the business, have to be dragged kicking and screaming to a slightly better state by the minority who do. It won't last long (hence the continued twitter examples).
- ano88888 3y ago
  strange.using messenger app daily and never encounter any issue.
- AbsoluteCabbage 3y ago
  The point is that performance is so important to customer satisfaction and the bottom line that even a horrendous company like facebook would begrudgingly spend time and money to improve performance. If anything your comment reinforces his point.
  - horsawlarway 3y ago
    I can rephrase your sentence as: The point is that performance is so far down the list of things to prioritize that [company of choice] made it to [revenue] without having to care at all about it. It wasn't worth them focusing on until they had already acquired a very large marketshare. Only when already large and successful did the scope and complexity of their system impact performance enough to bother focusing on it, at which point they begrudgingly did. ---- Performance is simply a requirement for the product, some products require very good performance, some don't. Where you fall is a discussion to be had, because like ALL product requirements, it comes with a cost to develop and maintain. I don't drive an F1 car to the grocery store. I don't take my minivan to the track. The article above is bad - it treats a conversation about product requirements as an antagonistic space, where voices that may prioritize a feature other than performance aren't making judgements about how to allocate limited resources, but "excuses"... Worse - he cherry picks the most extreme take of those value judgements for his examples as easy targets to attack.
Animats 3y ago
Do you want to reduce your compute costs, or improve the user's perceived response time? Those are quite different problems. They're approached in very different ways.
JanneVee 3y ago
Well then further excuses come flying on why he is wrong again, and how performance doesn't matter to them. I have a legacy code base that I until recently worked with that domain specific Business Intelligence product built with SOLID principles. 20% of the processes CPU usage was for the GC, whenever you made a dump of the process it was 99% fragmentation of memory and pointer chasing through object trees every time. The customers complain about performance because they aren't getting their data in a timely manner and not getting the savings they were promised with the product. It is not a game but it does math on a bunch of data and there is a requirement to do it in a timely manner, which it is failing. Stop coming with excuses that it is niche, if you are doing anything useful to the data then you might want to pick up on Caseys advice.
- kaba0 3y ago
  And what exactly is that advice? Most programs are not game engines operating upon millions of entities with the same basic logic. The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently. Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages.
  - JanneVee 3y ago
    The advice is that consider memory layout when doing repetitive operations on your data. > The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently. No my example isn't a complaint on it using GC language. It is that it is loading and unloading business entities multiple times and doesn't take into account that operations could be done linearly much faster instead of hunting a field in object graph to accumulate in various ways. That is how e.g. numpy works if you make sure that the array you do your operation on is of the same type. > Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages. Yes following the ideas of what is considered SOLID principals. The objections isn't managed vs. unmanaged it is going full OO and not giving a shit about how computers work.
tomgp 3y ago
I’ve never seen anyone make the argument that performance is not important. I have seen people make the argument that performance is less imprtant than some other property (maintainability, extensibility, legibility etc. etc.) given certain aims and constraints.
- kjksf 3y ago
  Everything cannot be important. If you say that maintainability, extensibility, legibility etc. etc. is more important than performance then you are, in fact, saying that performance is NOT important. But those are mostly just the excuses that Casey is ranting about. There's nothing inherent in writing fast code that makes is not maintainable, not extensible, not legible not etc., not etc. In fact, the things that make code less maintainable and less legible are often the things that make is slower. Things like replacing new Foo() with FooFactory and replacing FooFactory with FooFactoryProvider and then encoding logic in XML files. Those are things that people do way more often that writing SIMD intrinsic. And there are those who way overemphasize supposed "legibility" over performance. In JavaScript land you get legion claiming that the world will end if they rewrite unnecessarily slow map(), filter(), forEach() etc. with a regular for loop.
- AbsoluteCabbage 3y ago
  That's literally one of the excuses he debunks. Case in point: it is such a large factor on customer satisfaction and the bottom line that large companies such as Facebook would spend significant sums of time and money improving performance.
  - tomgp 3y ago
    Yes, clearly in that case performance is important but is performance always the most important thing? I read the article and didn't find it convincing, I would argue that it's clearly not always the most important thing. This isn't an "excuse" it's a calculation that teams make, the author feels that people make the tradeoff at the wrong point but instead of making that argument he frames decisions not to prioritise performance as "excuses" which is bullshit. There's always more performance optimisations one can make and there always comes a point where it just doesn't make sense to do so for all manner of reasons. A trivial example: I have a script which downloads a few thousand GIS Shape files and converts them into geoJSON. It runs automatically once a month, usually whilst I sleep. A run takes about 5 minutes at the moment but there are a couple of things I could do to make it run in a fraction of that time but then the script would be two or three times longer and more complex, and I'd have to spend a couple of hours writing and testing code, (there'd also be some edge cases that I'd need to account for which the current setup allows me to ignore). I judge that to be a waste of time which would make anyone who has to take ownership of this script in the future's life more difficult. So that's my "excuse" and I'm sticking to it.
hoseja 3y ago
Facebook seems(ed ~10 yrs ago) like such a great engineering company. Too bad they make Facebook.
DeathArrow 3y ago
I wonder what Casey thinks about using Javascript frameworks like Electron or Mobile Angular UI for everything, including desktop apps and mobile apps.
denieus 3y ago
I agree performance is important and we should aim to improve it if we can, but the cost x benefit of these improvements must be evaluated. It's easy to believe we're making huge gains when there's actually little gain added overall. Two examples: 1. I've seen people mentioning that following good programming practices make the code slower, and by removing them you can have improvements around 40%. That sounds like a great number, until you realize the real bottleneck are other things (e.g. database queries, network latency, etc). When you calculate the overall improvement for the request, the gains are negligible. 2. There are some frameworks that market themselves as crazy fast: "If you use us your app will boot almost instantaneously!". Looks cool, until you realize that a good pipeline will gradually rollout a new version and this will take time. Usually it comes with monitoring the new version for a while and then after it's deemed healthy we switch versions completely. Now instead of waiting a few minutes + 10 seconds, you will wait only a few minutes, which doesn't make much difference. Performance gains will come with tradeoffs and, before committing to that, it's a good idea to evaluate what are the real benefits of doing the changes we're planning to do.
joseph_grobbles 3y ago
I once was brought into a team that fervently bought into the "hotspot" argument, blustering ahead under the notion that performance was tomorrow's problem where someone would spend a day with a profiler and it would all be fixed. In reality their project was death by a thousand...neigh million or billions...of cuts. Poor technology choices. Poor algorithm choices. Incompetent usage (e.g. terrible LINQ usage everywhere, constantly). This was the sort of project where profiling was almost impossible because any profiling tool barfed up and gave up at every tier. Profiling the database was an exercise in futility. Profiling the middle tier was a flame graph that was endless peaks. Profiling the front-end literally crashed the browser. I ended up having to modify Chromium source to be able to accurately get a bead on how disastrously the Angular app was built. This is common. If performance doesn't matter to a team, it will never be something that can be easily fixed. Maybe you can throw a huge amount of money at the problem and scale up and out to a ridiculous degree for a tiny user base, but making an inefficient platform efficient is seldom easy.
cratermoon 3y ago
The author doesn't really debunk any of what he calls "excuses" when he jumps immediately to Facebook, then Twitter, Uber, Netflix, and Microsoft. Those are companies with huge scale problems. He even says as much, near the end, claiming "the evidence against these excuses comes from some of the largest and most financially valuable companies in the world". Nobody is arguing that performance doesn't matter at those scales. Here's the author's five points, and how at least one of the examples he gives proves the reasons. No need. These companies operate on the leading edge of hardware performance, on purpose. They can't just go out and buy faster hardware, it doesn't exist. Google even builds their own, just to optimize for their uses. Too small. Again, at the scale of Facebook or Netflix, a 5% performance gain translates to an enormous advantage, which leads directly to the next point. Not worth it. Here again, we're talking about saving millions of dollars but only because the systems are so enormous. Niche. Facebook, Twitter, Netflix, and Uber's performance needs are a niche of their own. Hostpot. Here we can get to a specific example the author quotes. "Cutting back on cookies required a few engineering tricks but was pretty straightforward; over six months we reduced the average cookie bytes per request by 42% (before gzip). To reduce HTML and CSS, our engineers developed a new library of reusable components (built on top of XHP) that would form the building blocks of all our pages." So their Facebook does have a hotspot, it just happens that it's a very large spot on a colossal size system. Finally, the author says, "If you look at readily-available, easy to interpret evidence, you can see that they are completely invalid excuses, and cannot possibly be good reasons to shut down an argument about performance." I'm still looking for the evidence.
draw_down 3y ago
Is there no performance gain that’s small enough to be not worth it? If there is, then we’re just haggling over where the line is. If there truly isn’t, well, enjoy that! Because you won’t be shipping.
li4ick 3y ago
You know, reading these comments is absolutely hilarious. It's because of this and that, hundreds of excuses that dance around the simple truth: you're an incompetent programmer. That's it. No, you're not capable of making something twice as fast if only you cared or had the time. If you haven't done it as a constant exercise, you can't.
- asynchronous 3y ago
  The sentiment here and in most of industry is: “and that doesn’t matter.”
  - alex7734 3y ago
    In some sectors software quality truly does not matter beyond a certain point (and the bar is really low). The customer very rarely has the resources or the time to try all the alternatives, especially if switching afterwards is expensive so if your marketing is good and your product works and has the needed features, you get sales, and if it's not, you don't. Is it terrible that the app takes minutes to add a few thousand numbers? Of course it is. But it does not matter, the customer is used to software being terrible so he won't waste the time and money to switch to another software that is probably also terrible.
  - lesuorac 3y ago
    The general problem though is that "and that doesn't matter" isn't backed up by anything but gut feeling. When FB dove into the numbers, they found they could save 50% in hardware costs. That _does_ matter. It may be the case that your company has a 20k server rack to support 20M of sales and so it doesn't matter. But unless you've actually looked up the cost you shouldn't be making the claim because it's unbacked; it's just a bad faith arguement.
    - mavhc 3y ago
      Also depends mostly on your size. The cost to rewrite your system to be twice as fast is about the same if you have 20k of hardware costs or 20M, but one saves 10k and the other 10M. Only one offsets the cost of the programmers
theptip 3y ago
Ironically, this article sets up a strawman of people who claim performance NEVER matters, and then rants long windedly in a tone suggesting it always matters. My general principle: don’t give uni-directional advice when optimizing a u-shaped loss function. I usually find that those failing to advocate the nuanced position “you can spend too much AND too little time on perf, here is how to prioritize” are not adding useful information to the conversation. The truth is, most startups don’t need to worry much about perf. It’s a feature that your customers don’t usually ask for at first. At the other end of the scale, giant companies invest huge sums in taming performance. And your own situation will have more parameters than just that one simplified spectrum. Measure ROI honestly, and prioritize accordingly!
- Hikikomori 3y ago
  He isn't saying that you should spend time on optimising your code, rather that you should write performant code from the beginning by not giving the cpu more work than it needs to do to perform the task, typically by avoiding OO paradigms, followings things like SOLID, etc. Then if needed, real optimisation can handle the hotspots in your code and is typically something that only a few of an orgs best engineers can do properly, but if everything is slow then its an insurmountable task for these few engineers. If you want to understand what he means with this watch the refterm series. https://www.youtube.com/watch?v=pgoetgxecw8 https://www.youtube.com/watch?v=pgoetgxecw8
- gizmo 3y ago
  The issue is that there is no need for software to be incredibly slow by default. This used not to be the case. Back in the day you just wrote straightforward code in C (or C++) and you got good performance for free. It makes sense that people want to use a programming language that allows for rapid prototyping if they want to get V1 out of the door. That's fine. But that doesn't explain why languages like Python and Ruby are incredibly slow in the first place. These are 20 year old languages that people have invested a ton of effort into. And despite a full decade of trying to make the languages faster they're still slow to point of near unusability. These languages, along with PHP and Javascript, were designed by people who paid zero attention to the performance consequences of the design decisions they made. Today, after industry having spend untold billions (yes, with a B) on trying to get the performance to an acceptable level getting anything written in these interpreted languages to run at an acceptable speed is painful. We are collectively wasting untold hours because our programming environments are bad, the languages are bad, the libraries we use are bad, and the standards are bad. This is not to say it's all hopeless, but we could be doing a lot better if we didn't focus so much on "moving fast and breaking things" and pursued quality instead.

141 points by brhsagain 3y ago

deleted 3y ago

[deleted]

leoncaet 3y ago

Casey does it again.

naught00 3y ago

What is funny is many of the exact excuses were made in this thread: https://news.ycombinator.com/item?id=34966137 https://news.ycombinator.com/item?id=34966137

AbsoluteCabbage 3y ago
yeah, and now people here are making the new excuses we already predicted lol

db48x 3y ago

> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 MBs off the total machine code size and bought us enough runway to keep development going. Wow. You know you have made bad choices when you need a desperation play like that.

typon 3y ago

I kind of wish Casey would stop making these videos. I'm being completely serious when I say that this kind of knowledge and mindset gives you pretty much a superpower in the industry. Everywhere I go people think I am amazing, even though I just know a few things about performance and low level programming (like, enough to be amazing for "mere mortals", but someone at RAD would snicker at my knowledge). If the industry doesn't care, it's simply a competitive advantage for you as an individual.

db48x 3y ago
I know what you mean; I’ve made a lot of money by cleaning up other people’s messes. Still, I’m pretty sure I’d prefer a world where software was just better overall and there were fewer messes.
- jiggawatts 3y ago
  I had a two year stint where I ran around doing "database optimisation" projects that were a week of billable time at consulting rates, and my entire "toolkit" was to simply flip three boolean flags from bad to good values. Now in the public cloud my gimmick is to switch VMs over to current-generation AMD EPYC models, and then collect my pay-check for the "performance tuning". The fruit is hanging so low that I'm trampling over it. There's no actual fruit picking going on.
  - runlaszlorun 3y ago
    I’ve always enjoyed perf testing and optimization but figured it’d be tough to reliably find clients staying in that niche. Any tips?
    - jiggawatts 3y ago
      I never got enough of this type myself, they were just random gigs between other types of project work. IMHO, the IT industry as a whole has internalised wastefulness and accepted poor performance because there's an expectation that "CPU upgrades will fix it". Over five decades that was true, it's just that few organisations have adapted to the new reality of only ~20% uplifts in single-thread performance over the last decade or so. In principle, there ought to be many more optimisation projects, but unfortunately they're still rare.
  - db48x 3y ago
    My favorite is to raise the Postgres work_mem value until their queries stop using disk buffers.
SPBS 3y ago

tmtvl 3y ago

I like the positive message he leaves us with: the barrier to entry for good performance is as low as it's ever been. You can get good performance without needing hand-coded assembly SIMD vector optimizations.

kragen 3y ago
also tho the barrier to entry for hand-coded assembly simd vector optimizations is as low as it's ever been

paul_grisham 3y ago

Obviously performance matters, it's just a question of opportunity cost. The article just takes some absolute statements about performance not mattering and argues against them, which is obvious and pointless. They ignore that the benefit/cost ratio for Facebook's performance improvements is vastly different to that of the average software company with tens of employees and some SaaS CRUD app. Their own example of Uber's app rewrite due to their previous architecture not being good enough is an example against their point - they got to where they were with imperfect architecture, then later fixed it once the opportunity cost tradeoff made sense.

db48x 3y ago
Did you read the same article I did? He explicitly states exactly that: > For example, one argument would be that the evidence I’ve presented here is consistent with a strategy of quickly shipping “version one” with poor performance, then starting work on a high-performance “version two” to replace it. That would be completely consistent with the evidence we see. > But even if that turns out to be true, it still means programmers have to care about performance! It just means they need to learn two modes of programming: “throw-away”, and “performant”. There would still be no excuse for dismissing performance as a critical skill, because you always know the throw-away version has to be replaced with a more performant version in short order. > That kind of argument is great. We should have it. What we should not have are excuses — claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle, so developers simply don’t have to learn about it.
- paul_grisham 3y ago
  > But even if that turns out to be true, it still means programmers have to care about performance! Who is he even talking to at this point? He's arguing that performance has more than 0% relevance to software, which no-one on disagrees with. >claims there is no argument to be had, and that performance somehow won’t matter anywhere in a product lifecycle No-one says performance never matters. People disagree on what performance merits optimization. A 5s to 1s page load improvement is massive, a .5s to .1s improvement starts hitting diminishing returns in user experience.

alphanumeric0 3y ago

"With so much evidence refuting the five excuses, hopefully it is clear that they are ridiculous. They are completely invalid reasons for the average developer, in the common case, to dismiss concerns about software performance. They should not be taken seriously in a professional software development context." I believe the average developer should care somewhat about performance, and depending on their industry they might need to care a lot, but I'm not so convinced for the average case. The average developer is not working on FAANG-sized codebases. Also, I'd imagine any large systems built up over the years that are refactored would likely see great performance gains. That's just the nature of long-term software.

magicalhippo 3y ago

There's often a lot of low hanging performance fruits when writing code. Need to loop over some order lines and find distinct article numbers? Use a hash-based set with O(1) access, not just a list which will have O(n). If not you'll end up writing an O(n^2) routine for no good reason, which will work swimmingly on you 10 line test order and cause grief in production. I don't think a lot about performance most of the time, just enough to try to avoid silly stuff.

kuvlopa 3y ago
Even better; let the database handle it if you have one, calling tons of extra data down the pipe is very slow compared to even slow iteration, and even the slow iteration is often fast if you only pull relevant data
citrin_ru 3y ago
O(1) hash vs O(N) array loop is a bit more complicated - for small N linear search can be faster - no need to compute a hash function, if data stored directly in the array (instead of pointers) then there will be less CPU cache misses. Exact threshold can be determined in a benchmark but the key here here is that N should be bounded - if a user can manipulate N to be huge and you have O(N^2) somewhere in your code it is not only performance but also a potential security problem - a DoS vector. IMHO the key to not writing silly code is to have a good mental model of how your code works and how it fits into a bigger picture which includes OS, hardware, network and other services (like a DB). This model should be updated from time to time based on benchmarks and metrics from production services but once you have this knowledge it takes almost no additional time to apply this in practice and even if you'll be taking shortcuts to save development time you'll be better aware of this tradeoff.
- magicalhippo 3y ago
  Yeah I was thinking in a CRUD setting, where I live, where you won't be doing the operation too often. In that case the inefficiency for small N is a non-issue vs the pain of a O(N^2).
- never_inline 3y ago

globalreset 3y ago

Scale / data set size tends to expose performance issues. But that's a good problem to have. Yes, it would be great if all developers could write performant code, but let's face it - there's only so many hours in a day and days in a week. Developers already struggle to keep up with all the required knowledge. It's not that people don't want to be competent. We're building more and more complex things while expanding the number of people employed building software which means average skill level is probably slightly decreasing.

HideousKojima 3y ago
>It's not that people don't want to be competent. There are plenty of devs that are just there for the paycheck and have minimal passion or interest in learning more.
- hikawaii 3y ago
  It's often not about learning more, the techniques and methods in Casey Muratori's streams are often very well understood and even taught in school. Dismissing people as just being there for the paycheck - that's you and every other developer. The number of people who produce useful OSS projects because they just like the project and aren't filling out resume points to hope bigtech senpai notices them is astronomically small, and usually the sort of political project nobody wants to touch.
- globalreset 3y ago
  True, but it's normal in many professions that lots of people working in it are not passionate and industry just deals with it. We can't expect that average software developer is going to be passionate and spending their weekends honing every-increasing set of skills: technologies, platforms, tools, etc.
  - eastbound 3y ago
    Can’t we? The entire startup ecosystem is based on extracting from the economy people who are motivated, do their work efficiently and learn high skills, and giving them millionaire-or-bankrupt stakes to ensure they’re committed.

charlie0 3y ago

Starts off with premise of why performance matters and why you should care. Proceeds to only top 10 biggest tech companies as examples. Are you working in a FAANG? If not, this article does nothing to convince me that I should care about performance (even though I kinda do).

readthenotes1 3y ago
It convinced me that there must be scads of failed attempts that focused on performance first.
omnicognate 3y ago
He's a games programmer. He's using examples outside his industry to illustrate that his points aren't specific to games (excuse 4). What industry would you like him to talk about?
- kragen 3y ago
  smes doing hourly contract crud development startups
  - vgatherps 3y ago
    To be fair, performance is usually part of the “does it work in real life” question, it’s just that doing sensible things is probably good enough. The crud app at your startup probably doesn’t need to worry about cutting optimizing every last millisecond off of page load or scaling across some k8s cluster or whatever. But I’ve stopped using (or never seriously tried) otherwise good services because of flow-stopping performance problems i.e. multi-second page loads, stuttering pages, battery drain, etc.
    - kragen 3y ago
      i'm not the one you need to convince, i'm just saying what marketing collateral i need to convince the skeptical: specific, detailed stories from those sectors, ideally at least somewhat verifiable (not a pseudonymous comment your wife's cousin read on hn) but specific and detailed goes a long way

klodolph 3y ago

Also note that “Facebook reduces the storage it needs to store user photos by 50%” is something that contributes, like, nine-figure dollar savings to the bottom line. Facebook does all sorts of crazy stuff to improve performance–I don’t think many people who work there understand how the javascript bundler works, for example. It’s a lot more than something like WebPack.

phendrenad2 3y ago

> a situation that sometimes happens does not support the use of a statement as a blanket excuse I feel like pointing this out has become table stakes in any conversation anywhere. Thinking of printing it out on a card and carrying it around.

devjab 3y ago

Performance matters, but I’m not sure people are saying that it doesn’t. I’m not a performance “first” type of programmer by any means, but I always keep it in the back of my mind, what I also tend to keep in the back of my mind is cost. I recently “inherited” a couple of back-end services when a developer left our company. It turned out that the code was terrible and that they haven’t used, any, or our helper tools. Since we use Typescript everywhere ignoring our quite opinionated and a little fascist linter rules is almost impossible, but the developer in question had the authority to turn it off, which they did and in doing so shot themselves completely in the foot. The back-end services were developed in JavaScript more than Typescript and since both our linters and usual teat pipelines were disabled, and since it’s software that has been developed over almost a years worth of changes, it was just horrible. We’re talking loops comparing values, that were probably there once but are now just sorting things as undefined === undefined kinds of terrible. The performance was also atrocious. Basically what the service did was gather info on a couple of thousand projects and link them with tens of thousands documents in Sharepoint, but because it was build wrong, it wasn’t pulling the correct documents and it was taking 5-10 minutes each run time. It’s now running at around 10 seconds for its complete run time. Which is a massive performance improvement, and it’ll be even better once I finish building the cashing. So you might think I’m inclined to agree with the article, but I didn’t rewrite it because of its poor performance, I rewrote it because it didn’t work correctly and the performance gains were simply a happy “coincidence”. This is because the performance didn’t really matter. Yes, it was costing us at most $77 a over our 3 year Azure contract, but the time I spent rebuilding it cost the company almost exactly $1500. Those $1500 were well spent because it wasn’t working, but would they have been well spent in terms of performance? Not really. That being said, it wouldn’t take a lot of those services to become expensive, so it’s not like the author is really wrong either. It’s just that I’m confused with whom he is arguing.

camgunz 3y ago
The points he lays out at the beginning of his post are basically Martin Fowler's (and thus 90% of enterprise engineers') position in Refactoring. Fowler was probably responding to engineers making code inscrutable in the name of (dubious claims of) performance, and Muratori here is responding to the ensuing overreaction.

tomas789 3y ago

One more datapoint here. Good performance helps avoiding scaling problems. One example is an API I recently wrote. I was initially in Python and performance was OK I guess. I had to take care of horizontal scaling, how to manage the infrastructure, a bit of distributed scaling and all the other goodies that come along. By rewriting it in Rust and caring about performance I was able to run it on a single machine (which is ok for that product). That saved me quite some hustle. And my average response time went from 300ms to 29ms. If the app would scale a lot I would still need to go into all that trouble but at this early stage the benefits of really simple infrastructure is immense.

musicale 3y ago
Rewriting your C++ app in Python can help it scale from 1000 servers to 100,000 servers without an increase in total throughput (although latency may increase.) It can also scale your power usage up from 1KW to 1000KW. If you have a datacenter that is 99% idle, you may be able to maximize utilization by switching to Python.

ftxbro 3y ago

So the biggest tech companies in the world, now that they are huge, are making their programs faster. OK. To me, that says maybe the reason they beat their competition to becoming huge, was that they wrote a lot of features quickly and didn't nitpick about performance. Maybe the ones that wrote efficient high performance code from the start stayed small or were out-competed.

simula67 3y ago
When I was starting out using the web, one of the reasons for using Google over competition was that it was fast. The home page loaded fairly quickly, and the results were instantaneous. Google spent a lot of time optimizing for performance during its early years and they were explicit about it.
- yonixw 3y ago
  The problem with this argument is that there are many contradicting sample points. Like chrome. On one hand, Google created the JS-Engine V8 which finally made the web fast. But on the other hand, did nothing to fix the RAM issues that came with that until recent years. So I see it as "Let's do performance only when it's an advantage for us". Which all of the OP examples falls to. OP want to conclude from that that performance=advantage so everyone need to work on it all the time (because who will ignore advantages?), but all we see is if(performance=advantage){refactor}. Which does not support his conclusion. Example cases are startups, monopoles, high cost (like refactoring bank code) etc. which have other silos of advantage.
AbsoluteCabbage 3y ago
you realize that you are literally parroting talking points that we have already predicted you would come up with as additional excuses, right?
- ftxbro 3y ago
  I mean I'm not parroting anything, but OK I will also agree that this isn't an especially mind blowing point that I was making. Probably a lot of other people would make that same point too!

yen223 3y ago

It bothers me that all the examples provided shows that companies care about performance ("Company $X made their app 20% faster"), but none of them talks about the impact of caring about performance ("Company $X made their app 20% faster, and saw a 200% increase in revenue as a result")

kjksf 3y ago
Casey obviously can't know what financial impact of those changes were at Twitter, Facebook or Microsoft. But Ockham Razor tells us they did spend significant effort on performance because they had a reason to do it.

xupybd 3y ago

The author fails to make the business case for their arguments. Performance tuning costs money. Don't spend money with no expected ROI. Yes at Facebook with end user facing software it is crucial to get performance right. If you're running payroll at a 3 person company it doesn't matter if the software is inefficient. Most of the time it's Excel, and that is not the most efficient way to do those calculations. But it's not worth investing in a better solution until processing payroll is a problem.

School-Cotton 3y ago

Absolutely hilarious and ridiculous to claim that Facebook cares about iOS app performance. In reality they have a few dozen people who care making heroic technical fixes to mitigate the damage of thousands of people adding mountains of mediocre crap to the apps. If you care about performance from the beginning, you would never even get to the point where you’re saying “oh crap, our messenger app can barely run, let’s rewrite it in C”.

Jach 3y ago
Late to the party but this is the same point of disagreement I had with the video, though overall it was fine. Why do these performance efforts and rewrites at the companies he listed take years? How complex is the uber eats app anyway? Well, it's because most engineers at those (and other) companies do not care or know about performance, and when it actually matters to the business, have to be dragged kicking and screaming to a slightly better state by the minority who do. It won't last long (hence the continued twitter examples).
ano88888 3y ago
strange.using messenger app daily and never encounter any issue.
AbsoluteCabbage 3y ago
The point is that performance is so important to customer satisfaction and the bottom line that even a horrendous company like facebook would begrudgingly spend time and money to improve performance. If anything your comment reinforces his point.
- horsawlarway 3y ago
  I can rephrase your sentence as: The point is that performance is so far down the list of things to prioritize that [company of choice] made it to [revenue] without having to care at all about it. It wasn't worth them focusing on until they had already acquired a very large marketshare. Only when already large and successful did the scope and complexity of their system impact performance enough to bother focusing on it, at which point they begrudgingly did. ---- Performance is simply a requirement for the product, some products require very good performance, some don't. Where you fall is a discussion to be had, because like ALL product requirements, it comes with a cost to develop and maintain. I don't drive an F1 car to the grocery store. I don't take my minivan to the track. The article above is bad - it treats a conversation about product requirements as an antagonistic space, where voices that may prioritize a feature other than performance aren't making judgements about how to allocate limited resources, but "excuses"... Worse - he cherry picks the most extreme take of those value judgements for his examples as easy targets to attack.

Animats 3y ago

Do you want to reduce your compute costs, or improve the user's perceived response time? Those are quite different problems. They're approached in very different ways.

JanneVee 3y ago

Well then further excuses come flying on why he is wrong again, and how performance doesn't matter to them. I have a legacy code base that I until recently worked with that domain specific Business Intelligence product built with SOLID principles. 20% of the processes CPU usage was for the GC, whenever you made a dump of the process it was 99% fragmentation of memory and pointer chasing through object trees every time. The customers complain about performance because they aren't getting their data in a timely manner and not getting the savings they were promised with the product. It is not a game but it does math on a bunch of data and there is a requirement to do it in a timely manner, which it is failing. Stop coming with excuses that it is niche, if you are doing anything useful to the data then you might want to pick up on Caseys advice.

kaba0 3y ago
And what exactly is that advice? Most programs are not game engines operating upon millions of entities with the same basic logic. The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently. Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages.
- JanneVee 3y ago
  The advice is that consider memory layout when doing repetitive operations on your data. > The usual application where GC is most relevant is connected to 10s of services over some network protocol, and often has to load each business object/entity it operates on into memory to check permissions, sometimes even issue another network call, etc. Loading that entity from a list of pointers is absolutely no bottleneck in such a setting, at all - the way you might speed up the app is doing the work on the database itself, which is already written in some low-level language, efficiently. No my example isn't a complaint on it using GC language. It is that it is loading and unloading business entities multiple times and doesn't take into account that operations could be done linearly much faster instead of hunting a field in object graph to accumulate in various ways. That is how e.g. numpy works if you make sure that the array you do your operation on is of the same type. > Your anecdote is just a badly written application, I have plenty examples of very performant ones written in managed languages. Yes following the ideas of what is considered SOLID principals. The objections isn't managed vs. unmanaged it is going full OO and not giving a shit about how computers work.

tomgp 3y ago

I’ve never seen anyone make the argument that performance is not important. I have seen people make the argument that performance is less imprtant than some other property (maintainability, extensibility, legibility etc. etc.) given certain aims and constraints.

kjksf 3y ago
Everything cannot be important. If you say that maintainability, extensibility, legibility etc. etc. is more important than performance then you are, in fact, saying that performance is NOT important. But those are mostly just the excuses that Casey is ranting about. There's nothing inherent in writing fast code that makes is not maintainable, not extensible, not legible not etc., not etc. In fact, the things that make code less maintainable and less legible are often the things that make is slower. Things like replacing new Foo() with FooFactory and replacing FooFactory with FooFactoryProvider and then encoding logic in XML files. Those are things that people do way more often that writing SIMD intrinsic. And there are those who way overemphasize supposed "legibility" over performance. In JavaScript land you get legion claiming that the world will end if they rewrite unnecessarily slow map(), filter(), forEach() etc. with a regular for loop.
AbsoluteCabbage 3y ago
That's literally one of the excuses he debunks. Case in point: it is such a large factor on customer satisfaction and the bottom line that large companies such as Facebook would spend significant sums of time and money improving performance.
- tomgp 3y ago
  Yes, clearly in that case performance is important but is performance always the most important thing? I read the article and didn't find it convincing, I would argue that it's clearly not always the most important thing. This isn't an "excuse" it's a calculation that teams make, the author feels that people make the tradeoff at the wrong point but instead of making that argument he frames decisions not to prioritise performance as "excuses" which is bullshit. There's always more performance optimisations one can make and there always comes a point where it just doesn't make sense to do so for all manner of reasons. A trivial example: I have a script which downloads a few thousand GIS Shape files and converts them into geoJSON. It runs automatically once a month, usually whilst I sleep. A run takes about 5 minutes at the moment but there are a couple of things I could do to make it run in a fraction of that time but then the script would be two or three times longer and more complex, and I'd have to spend a couple of hours writing and testing code, (there'd also be some edge cases that I'd need to account for which the current setup allows me to ignore). I judge that to be a waste of time which would make anyone who has to take ownership of this script in the future's life more difficult. So that's my "excuse" and I'm sticking to it.

hoseja 3y ago

Facebook seems(ed ~10 yrs ago) like such a great engineering company. Too bad they make Facebook.

DeathArrow 3y ago

I wonder what Casey thinks about using Javascript frameworks like Electron or Mobile Angular UI for everything, including desktop apps and mobile apps.

denieus 3y ago

I agree performance is important and we should aim to improve it if we can, but the cost x benefit of these improvements must be evaluated. It's easy to believe we're making huge gains when there's actually little gain added overall. Two examples: 1. I've seen people mentioning that following good programming practices make the code slower, and by removing them you can have improvements around 40%. That sounds like a great number, until you realize the real bottleneck are other things (e.g. database queries, network latency, etc). When you calculate the overall improvement for the request, the gains are negligible. 2. There are some frameworks that market themselves as crazy fast: "If you use us your app will boot almost instantaneously!". Looks cool, until you realize that a good pipeline will gradually rollout a new version and this will take time. Usually it comes with monitoring the new version for a while and then after it's deemed healthy we switch versions completely. Now instead of waiting a few minutes + 10 seconds, you will wait only a few minutes, which doesn't make much difference. Performance gains will come with tradeoffs and, before committing to that, it's a good idea to evaluate what are the real benefits of doing the changes we're planning to do.

joseph_grobbles 3y ago

I once was brought into a team that fervently bought into the "hotspot" argument, blustering ahead under the notion that performance was tomorrow's problem where someone would spend a day with a profiler and it would all be fixed. In reality their project was death by a thousand...neigh million or billions...of cuts. Poor technology choices. Poor algorithm choices. Incompetent usage (e.g. terrible LINQ usage everywhere, constantly). This was the sort of project where profiling was almost impossible because any profiling tool barfed up and gave up at every tier. Profiling the database was an exercise in futility. Profiling the middle tier was a flame graph that was endless peaks. Profiling the front-end literally crashed the browser. I ended up having to modify Chromium source to be able to accurately get a bead on how disastrously the Angular app was built. This is common. If performance doesn't matter to a team, it will never be something that can be easily fixed. Maybe you can throw a huge amount of money at the problem and scale up and out to a ridiculous degree for a tiny user base, but making an inefficient platform efficient is seldom easy.

cratermoon 3y ago

The author doesn't really debunk any of what he calls "excuses" when he jumps immediately to Facebook, then Twitter, Uber, Netflix, and Microsoft. Those are companies with huge scale problems. He even says as much, near the end, claiming "the evidence against these excuses comes from some of the largest and most financially valuable companies in the world". Nobody is arguing that performance doesn't matter at those scales. Here's the author's five points, and how at least one of the examples he gives proves the reasons. No need. These companies operate on the leading edge of hardware performance, on purpose. They can't just go out and buy faster hardware, it doesn't exist. Google even builds their own, just to optimize for their uses. Too small. Again, at the scale of Facebook or Netflix, a 5% performance gain translates to an enormous advantage, which leads directly to the next point. Not worth it. Here again, we're talking about saving millions of dollars but only because the systems are so enormous. Niche. Facebook, Twitter, Netflix, and Uber's performance needs are a niche of their own. Hostpot. Here we can get to a specific example the author quotes. "Cutting back on cookies required a few engineering tricks but was pretty straightforward; over six months we reduced the average cookie bytes per request by 42% (before gzip). To reduce HTML and CSS, our engineers developed a new library of reusable components (built on top of XHP) that would form the building blocks of all our pages." So their Facebook does have a hotspot, it just happens that it's a very large spot on a colossal size system. Finally, the author says, "If you look at readily-available, easy to interpret evidence, you can see that they are completely invalid excuses, and cannot possibly be good reasons to shut down an argument about performance." I'm still looking for the evidence.

draw_down 3y ago

Is there no performance gain that’s small enough to be not worth it? If there is, then we’re just haggling over where the line is. If there truly isn’t, well, enjoy that! Because you won’t be shipping.

li4ick 3y ago

You know, reading these comments is absolutely hilarious. It's because of this and that, hundreds of excuses that dance around the simple truth: you're an incompetent programmer. That's it. No, you're not capable of making something twice as fast if only you cared or had the time. If you haven't done it as a constant exercise, you can't.

asynchronous 3y ago
The sentiment here and in most of industry is: “and that doesn’t matter.”
- alex7734 3y ago
  In some sectors software quality truly does not matter beyond a certain point (and the bar is really low). The customer very rarely has the resources or the time to try all the alternatives, especially if switching afterwards is expensive so if your marketing is good and your product works and has the needed features, you get sales, and if it's not, you don't. Is it terrible that the app takes minutes to add a few thousand numbers? Of course it is. But it does not matter, the customer is used to software being terrible so he won't waste the time and money to switch to another software that is probably also terrible.
- lesuorac 3y ago
  The general problem though is that "and that doesn't matter" isn't backed up by anything but gut feeling. When FB dove into the numbers, they found they could save 50% in hardware costs. That _does_ matter. It may be the case that your company has a 20k server rack to support 20M of sales and so it doesn't matter. But unless you've actually looked up the cost you shouldn't be making the claim because it's unbacked; it's just a bad faith arguement.
  - mavhc 3y ago
    Also depends mostly on your size. The cost to rewrite your system to be twice as fast is about the same if you have 20k of hardware costs or 20M, but one saves 10k and the other 10M. Only one offsets the cost of the programmers

theptip 3y ago

Ironically, this article sets up a strawman of people who claim performance NEVER matters, and then rants long windedly in a tone suggesting it always matters. My general principle: don’t give uni-directional advice when optimizing a u-shaped loss function. I usually find that those failing to advocate the nuanced position “you can spend too much AND too little time on perf, here is how to prioritize” are not adding useful information to the conversation. The truth is, most startups don’t need to worry much about perf. It’s a feature that your customers don’t usually ask for at first. At the other end of the scale, giant companies invest huge sums in taming performance. And your own situation will have more parameters than just that one simplified spectrum. Measure ROI honestly, and prioritize accordingly!

Hikikomori 3y ago
He isn't saying that you should spend time on optimising your code, rather that you should write performant code from the beginning by not giving the cpu more work than it needs to do to perform the task, typically by avoiding OO paradigms, followings things like SOLID, etc. Then if needed, real optimisation can handle the hotspots in your code and is typically something that only a few of an orgs best engineers can do properly, but if everything is slow then its an insurmountable task for these few engineers. If you want to understand what he means with this watch the refterm series. https://www.youtube.com/watch?v=pgoetgxecw8 https://www.youtube.com/watch?v=pgoetgxecw8
gizmo 3y ago
The issue is that there is no need for software to be incredibly slow by default. This used not to be the case. Back in the day you just wrote straightforward code in C (or C++) and you got good performance for free. It makes sense that people want to use a programming language that allows for rapid prototyping if they want to get V1 out of the door. That's fine. But that doesn't explain why languages like Python and Ruby are incredibly slow in the first place. These are 20 year old languages that people have invested a ton of effort into. And despite a full decade of trying to make the languages faster they're still slow to point of near unusability. These languages, along with PHP and Javascript, were designed by people who paid zero attention to the performance consequences of the design decisions they made. Today, after industry having spend untold billions (yes, with a B) on trying to get the performance to an acceptable level getting anything written in these interpreted languages to run at an acceptable speed is painful. We are collectively wasting untold hours because our programming environments are bad, the languages are bad, the libraries we use are bad, and the standards are bad. This is not to say it's all hopeless, but we could be doing a lot better if we didn't focus so much on "moving fast and breaking things" and pursued quality instead.