21 ms·

Performance bugs – the dark matter of programming bugs

129 points by Ono-Sendai 10y ago

radarsat1 10y ago
Reminds me of https://accidentallyquadratic.tumblr.com/ https://accidentallyquadratic.tumblr.com/
bluetomcat 10y ago
Back in 2001, Joel Spolsky coined "Schlemiel the Painter's algorithm" to illustrate the same class of inefficient programming techniques: https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_algorithm https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Pai... His particular example was with the standard C library function strcat where each new invocation traverses the enlarged string over and over from the beginning, in order to find the terminating NUL character and append the new part.
smnscu 10y ago
Already down. edit: up again now https://webcache.googleusercontent.com/search?q=cache:o4ftZJqeuhsJ:https://www.forwardscattering.org/post/49+&cd=1&hl=en&ct=clnk&gl=ro https://webcache.googleusercontent.com/search?q=cache:o4ftZJ...
- Ono-Sendai 10y ago
  Oh the irony :) Will try to fix. Edit: Back up, let's see how long it lasts now :)
  - yetihehe 10y ago
    Less than 15min ;) Edit: back up after further 10min. Seems like it moves in waves.
deleted 10y ago
[deleted]
pjc50 10y ago
This is one of the few areas where adding more levels of abstraction cannot help you fix the problem. It's remarkably easy for some library function or low level of the program to have O(n) performance that ends up getting called in a loop with the same input or output.
- gmfawcett 10y ago
  It's true that libraries can cause performance issues, but it's not a given. Using template metaprogramming libraries (like STL and Boost in C++) can result in clean, high-level code which has very good performance. Often the "loops" and other high-level constructs are optimized right out of the program. Also, link-time and whole-program optimization can remedy some performance issues across modules/libraries. [edit: not algorithmic problems, of course, LTO isn't going to rewrite your algorithms for you.]
  - tyingq 10y ago
    It does, in some situations, create a visibility issue. Very common in java. Using some higher level library like cxf, for example, for https connectivity to services. Unknown to you, and somewhat difficult to unearth (many layers down the rabbit hole to find the actual socket code, not in the cxf code), is that connection pooling doesn't work well, and every connection spawns more than one underlying object.
  - AstralStorm 10y ago
    STL or rather standard C++ library can result in clean code. Boost... rarely, especially the more advanced metaprogramming tricks there are esoteric. Neither typically provides highest performance. (I.e. boost ublas vs Eigen)
- hacker_9 10y ago
  Depends on the problem? Abstractions can usually make assumptions about the data, and thus be faster.
- wtetzner 10y ago
  > This is one of the few areas where adding more levels of abstraction cannot help you fix the problem. I'm not sure that's true. It is of course true that adding levels of abstraction often degrades performance, but I don't think it's necessary. In fact, programming languages themselves are abstractions. It's pretty well agreed at this point that a good C compiler will generate faster code than someone writing assembly. It's of course possible to write code that's at least as fast in assembly, but in practice, nobody would do it, because the program would be unmaintainable. You can make abstractions fast, and one of the benefits of doing so is that you can make your abstraction be the easy way of doing things. That means, by picking the easiest solutions, the people on your team are also using an efficient solution.
nateberkopec 10y ago
Personal blog authors - if you insist on not using Medium/similar big box blogging service, just host your blog as flat files on S3. It will never, ever go down when you hit number 1 on HN (as this poor author's site has).
- skarap 10y ago
  It wouldn't go down with flat files even if you hosted it on an arduino connector to your phone's tethered wifi.
  - toast0 10y ago
    How many tls handshakes per second can an arduino manage? Https everywhere isn't cheap.
- pat2man 10y ago
  You can also throw cloudflare in front of it for free.
- mrighele 10y ago
  An alternative is to cache request at the web server level so that successive requests will not be more heavy than serving static page. Caching just for a few seconds (some people call it microcaching [1]) will let your blog survive without issues. [1] https://www.nginx.com/blog/benefits-of-microcaching-nginx/ https://www.nginx.com/blog/benefits-of-microcaching-nginx/
maximilianroos 10y ago
Ironic that the site is down...
tcopeland 10y ago
At one point I was working on a utility to detect suboptimal sequence of method calls; canonical example is using "[1,2,3].select {|x| x > 1 }.first" rather than "[1,2,3].detect {|x| x > 1}". These can be performance issues, although the bigger win is in readability and communicating the developer intent. More details and examples here if you're interested: https://thomasleecopeland.com/2014/10/22/finding-suboptimal-api-usage.html https://thomasleecopeland.com/2014/10/22/finding-suboptimal-... I haven't worked on it much recently though because the problems it found weren't that significant. But I like the idea of runtime analysis for finding issues, especially in a dynamically-typed language.
draw_down 10y ago
"Should" is a tricky word.
base698 10y ago
In practice the majority of web app performance have more to do with latency of IO. People seem to forget that 4ms may seem fast to humans, but it's glacial to computers. Add 100 4ms calls to the DB and all the sudden your code is waiting for half a second if it does nothing else.
- AstralStorm 10y ago
  Bonus points that in a real real-time system (anything with a GUI) you might have only a 1 ms or less to handle an event in a bigger stream, e.g. dragging. This is why so much UI is or should be really asynchronous.
unsoundInput 10y ago
Even though I agree that >>insert method of std::map is roughly 7 times slower than it should be<< is bad, these kind of problems are not too hard to find and solve if they are actually problematic for your software. The most problematic performance issues I've come across were usually bad/premature optimizations that were not (correctly) validated against a simpler implementation as a performance baseline. Things like parallelism (multi-threading, web workers) or caching can absolutely tank performance if not done correctly. Plus they usually tend to make stuff more complex and bug-prone.
rosshemsley 10y ago
One of my favourite examples: A friend once sped up some code 90% by changing a list.size() == 0 to list.empty() yes, std::list::size() can be _linear_ in C++98. http://www.cplusplus.com/reference/list/list/size/ http://www.cplusplus.com/reference/list/list/size/
- logophobia 10y ago
  std::list is a linked list, so it makes sense that size() is linear. Only if you store the size separately, you get constant behavior (which the c++11 enforces), it's constant. Your friend probably could've sped his code up even more by using a std::vector. Linked lists have very bad processor cache behavior (not allocated in continuous memory), there are not a lot of things a list does better than a vector. Only if you prepend a lot, or insert in the middle of a list, it might have some advantages over a vector, even then, there're are better alternatives.
  - sqeaky 10y ago
    I have timed it on several implementations and posted (mostly on on reddit.com/r/cpp), but until you get several thousand items std::vector is just faster than std::list. Of course it varies from machine to machine, for example my 4th gen i7 could insert about 8,000 items before list finally got the same speed as inserting at the front of an std::vector and my AMD 8150 (The bulldozer thing way before ryzen) was about 3,500. I did similar tests comparing linear scans through an std::vector and lookups in a std::map or std::unsorted_map, with similar results, many thousands of items are required before dumb vectors got slow. My default rule is to use a std::array if the size is fixed or an std::vector if the size changes until I hit 1,000 items. Then look at the complexity notation of the operations I am doing and choose what containers to then to benchmark. Then I throw a couple microbenchmarks in the test suite and use whatever that tells me will be faster in my case. Sometimes I even go so far as to use ifdefs and typedefs to change the container for different builds. In practice I use a lot of std::vectors and I sort them a lot. Sorted vectors are very fast in the general case. In the few workloads I have with large numbers of items, even with millions of items, I use std::sort and std::binary_search with std::vector because they can be much faster than other associative containers. You do need to use operator< intelligently and that means either already passing in an overload for the comparator function or making a simple container class that wraps these concepts the way that makes sense for your workload.
dahart 10y ago
> Performance bugs are out there, lurking and unseen Is it a bug if it's unseen? Code that runs slower than it could in theory is literally everywhere. But is it worth paying the cost of the time to fix it, if you aren't experiencing any delays? I love the way Michael Abrash talks about optimizing. He emphasizes and re-emphasizes that optimization is for people not computers, and that if you aren't experiencing a delay, you're done. "In other words, high-performance code should ideally run so fast that any further improvement in the code would be pointless. "Notice that the above definition most emphatically does not say anything about making the software as fast as possible. It also does not say anything about using assembly language, or an optimizing compiler, or, for that matter, a compiler at all. It also doesn’t say anything about how the code was designed and written. What it does say is that high-performance code shouldn’t get in the user’s way—and that’s all." http://www.jagregory.com/abrash-black-book/#the-human-element-of-code-optimization http://www.jagregory.com/abrash-black-book/#the-human-elemen...
- vilya 10y ago
  I think that was true back when he wrote the words, but less so now. Even if you're not noticing a delay, code that completes quicker is generally code that uses less power and you can still notice the effect on your phones battery life - or on the size of your monthly bill from your data center.
  - dahart 10y ago
    Great point. Efficiency comes in different shapes. To respond, I'd say power & cost are human oriented metrics, and I take his point to be use a human oriented metric, for me it's as true now as it ever was.
  - datenwolf 10y ago
    Ah darn, I wrote along the same line an hour ago or so but got delayed in hitting the submit button until now. Yes, pretty much this.
- pdpi 10y ago
  "Unseen" is relative A performance issue that is hard to detect at 10 rps becomes an issue when you're trying to push 1000 rps. A performance issue in a mobile app (or a desktop app on a laptop) might be undetectable from a responsiveness or frame rate point of view, but burn through your battery. If you count memory footprint as a performance issue, thowr don't become apparent until you try to run more things concurrently, and find yourself thrashing when memory should be plentiful.
prohor 10y ago
The fun begins when you need to find performance issues in production, where you cannot really use profiler. Then you need to jump into APM tools. Unfortunately it seems there nothing really for that level for free, but take this for example: https://www.dynatrace.com/blog/code-level-visibility-for-node-js/ https://www.dynatrace.com/blog/code-level-visibility-for-nod...
- whyever 10y ago
  Can't you just use something like performance counters, like perf for Linux?
  - itsderek23 10y ago
    In my experience, it's very difficult to tie profiling data from generic profilers to specific requests, then to the specific lines-of-code triggering the problems. This is important because many performance conditions don't reveal themselves all of the time: for example, it's very common that an issue might only be a problem for your largest customers. The context is really important. Scout has a production-safe profiler for Ruby apps that builds on the wonderful StackProf gem that does this: http://help.apm.scoutapp.com/#scoutprof http://help.apm.scoutapp.com/#scoutprof
amorphid 10y ago
If performance matters, benchmarking the heck out of everything helps quite a bit! I like the style of benchmarks that focus on comparing iterations per second.
kazinator 10y ago
It's only a bug if the stakeholders that surround the program agree that there should be (or already is) in place a performance requirement which is not being met. Opportunities to uncover performance that could be improved and impose requirements against it are out there, lurking unseen. Only problem is you have to convince people to care on a case-by-case basis.
kevindqc 10y ago
> when I was working on the Doom 3 BFG edition release, the exactly predicted off-by-one-frame-of-latency input sampling happened, and very nearly shipped What is this predicted off-by-one-frame-of-latency input sampling ?
partycoder 10y ago
Performance issues come in all flavors... - CPU usage (complexity, cache misses...) - Memory usage (leaked resources like memory, connections and handles) - Excessive I/O (networking, files...) - Concurrency (contention, starvation, deadlocks, livelocks...) There's no one size fits all profiler. I find load testing in combination with profiling very useful to get an idea of how a system performs. Usually I keep injecting load until the system fails then find what failed. There are other approaches, like automated performance testing (writing tests with assertions on performance). e.g: https://scalameter.github.io https://scalameter.github.io
deleted 10y ago
[deleted]
bastijn 10y ago
I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud environments) is a real pain here. The google white paper on Dapper [0] and all its implementations [1][2][3] in one form or another are appearing but usually target the Web environments. That means that for my legacy stack with multiple nodes connected via networks of different kinds I have to profile on multiple systems, aggregate different log types manually, think of time syncs (not everything logs utc..) and so on. To be honest we just reached the point where we try to get a European subsidiary project rolling where we address this problem. The time lost to trace down unexplainable time gaps between first displayed result in app and request are weeks, and in multiple occasions months. Just because we have no easy way to profile our distributed ecosystem. After identifying who is to blame: network, db nodes, processing nodes, application logic, cache (invalidation); we can talk about fixing. For fixing of course it would be great if you could extract a meta model of the application and another of the hardware. Changes to those models and some nice prediction of impact would be the ideal end goal. Or, if we are wishing anyway, some nice DSE on those models to optimize would be even better. [0] https://research.google.com/pubs/pub36356.html https://research.google.com/pubs/pub36356.html [1] http://opentracing.io/ http://opentracing.io/ [2] http://lightstep.com/ http://lightstep.com/ [3] https://aws.amazon.com/xray/ https://aws.amazon.com/xray/
- pritianka 10y ago
  OpenTracing has libraries for C++ (though it needs work tbh) and Java which can be used for mobile. The idea is to cover your entire stack ...
  - bastijn 10y ago
    I know. But our stack is C#, Wcf, websockets with c++ where performance is required and some RestAPI/wpf/html5/typescript/javascript ui/clients. The stack is something grown the past 15 or so years (medical business). The existing options are not mature enough yet for us to be used, or not available yet on our platforms.
ptx 10y ago
One recent example of this is Synapse, the Matrix messaging server. Synapse was generally considered to be inefficient and a huge memory hog, so they have been experimenting with various ways to rewrite it in Go, saying[1] at the end of last year: "Synapse is in a relatively stable state currently ... we’re starting to hit some fundamental limitations of the architecture: ... python’s single-threadedness and memory inefficiency; ...; the fact the app papers over SQL problems by caching everything in RAM (resulting in synapse’s high RAM requirements); ..." But in January[2] it was discovered, during the investigation of another problem, that the vast majority of the memory usage was due to a bug: "It’s also revealed the root cause of why Synapse’s RAM usage is quite so bad – it turns out that it actually idles at around 200MB with default caching, but there’s a particular codepath which causes it to spike temporarily by 1GB or so – and that RAM is then not released back to the OS." I wonder if they would still have started the Go rewrite if it hadn't been for that bug. (It seems the new implementation has a better architechture and is substantially faster, so I guess it's still a good idea though, but maybe not as urgent as before?) [1] https://matrix.org/blog/2016/12/26/the-matrix-holiday-special-2016-edition/ https://matrix.org/blog/2016/12/26/the-matrix-holiday-specia... [2] https://matrix.org/blog/2017/01/06/synapse-0-18-6-is-out-please-upgrade-especially-if-on-0-18-5/ https://matrix.org/blog/2017/01/06/synapse-0-18-6-is-out-ple...
- Namrog84 10y ago
  This is a great example of why you can't take perf related issues intuitively or at face value. Profiling and in depth analysis is always needed. I wonder if this could have been more easily spotted earlier on if a better analysis was done.
- BurningFrog 10y ago
  So the analysis of why it was slow was just informed guesswork. One thing I keep learning again and again: You never know what the performance problem is until you've measured it.
- Arathorn 10y ago
  We would absolutely still be doing the go-rewrite: while fixing that bug improved the memory usage, it didn't help actual performance much. Meanwhile Dendrite (the rewrite) is currently benchmarking 2 orders of magnitude faster than Synapse. Unfortunately Synapse's architecture simply has many design efficiencies :)
wolfgang42 10y ago
Quote from the linked article by John Carmack[1]: > The fly-by-wire flight software for the Saab Gripen (a lightweight fighter) went a step further... Control flow went forward only. Sometimes one piece of code had to leave a note for a later piece telling it what to do, but this worked out well for testing: all data was allocated statically, and monitoring those variables gave a clear picture of most everything the software was doing.... No bug has ever been found in the “released for flight” versions of that code. When I started writing Arduino code, I independently re-invented this technique. I write my code with only fixed-size loops and no delay() calls (basically cooperative multitasking), making heavy use of finite-state machines to keep track of what should happen when. This style of code takes some getting used to, but its advantages are enormous: I see other people using delays and busy-loops and then struggling to get interrupts working to keep their device responsive, whereas I have a clean linear set of logic that can be easily understood and is free of strange race conditions and the like, and is guaranteed to keep responding to events. This style also makes it easy to literally eyeball performance: attach a bicolor LED (one that has e.g. red and green LEDs in the same package) and tell it to toggle colors once per loop. If the main loop is taking more than 8 milliseconds or so the LED will start to flicker visibly; if something pauses the LED will go solid and it's obvious the main loop is frozen. [1] http://number-none.com/blow/blog/programming/2014/09/26/carmack-on-inlined-code.html http://number-none.com/blow/blog/programming/2014/09/26/carm... (Edits for clarity and writing style.)
- dahart 10y ago
  One of the interesting side effects of this style is that you sometimes choose to let code run slower than it could, and choose to use more memory than you need. The point is (as you say) to make all loops & allocation as predictable as possible, so when you make them fixed size, you have to fix it on the worst case. Ideally there is no worst case, and the amount of work you do per frame or cycle is constant. But for anything non-trivial, chances are fixed allocations & loops aren't always needed, even though they're extremely useful. This does exactly what you say, in my experience: it makes performance eyeball-able, you can see where problems are. Code becomes so easy to reason about that debugging is easier and time is saved.
- RangerScience 10y ago
ericb 10y ago
I'm open sourcing a Java Selenium library with page performance assertions built in as part of my startup. It will let you do things like: test.config.setSLAMaxTime(2000); // set time in MS assertSLAMet(); // add this to any page It will also allow you to assert specific URL's, and window timings. You can then use these assertions to pass/fail builds. I also have some other nifty ideas on preventing performance regressions I'm building into this (smart benchmarking vs. previous). If anyone is interested, I'll be announcing it here when we open source it: http://signup.browserup.com/ http://signup.browserup.com/
- orf 10y ago
  > What if your Selenium tests had performance asserts for CI/CD Isn't this just... a single function? Like, how the heck do you build a startup around that. We do this as part of our Python selenium tests, I didn't realize it wasn't built in to be honest.
  - ericb 10y ago
    There's a lot more to it--this is just a tiny part we're giving away, not something we'll make money on. We do real-browser load testing in the cloud using the same scripts folks use for integration testing. Because they can also run locally, you can run them alongside your integration tests with each commit, and you always know you have working scripts. Scripts are easy to code and maintain because they can use page objects like functional tests. This means you can have versioned load test/monitoring scripts in your project that always work with a particular sha. When you deploy, you can start transactional monitoring immediately. It also means load testing is not a separate stage in your development lifecycle.
btilly 10y ago
I have seen too many of these to keep track of. But I will never forget once having to work with some Windows software written by someone else. He decided to parse a CSV file by reading the whole thing in memory, and then literally chopping off one character at a time..copying the rest of the string. It made what should be an instantaneous import of a fairly small file into a 20 minute ordeal. Simply reading in one line at a time would have been a massive speedup. I did not have access to change it. I asked him to fix it. I begged him to fix it. His response was to shrug and say, "Who cares? This runs as part of a batch process with nobody in front of it. It works fine." That may be, but I cared because I had to sit through it repeatedly while debugging certain misbehavior that arose in said batch process. Years later I was talking to an ex-coworker who stayed at said company after I left. He reported that that original developer after I left had to make a fix to said batch process. While doing so he had fixed the performance bug and was an instant hero among a number of other people who had to sit through the same import repeatedly over time. Yeah..it turns out that he had made the fix I'd originally asked. And nobody realized that he had caused the problem originally AND refused to fix it. That's one developer that I'm glad I will never have to work with again...
- dahart 10y ago
  That would be pretty frustrating and annoying! This reminds me of a story I heard about a lead game programmer who made a habit of pre-allocating an unused megabyte at the beginning of a project. The last week before ship, when the game would crash out of memory often and everyone was freaking out and trying to squeeze every last byte out of their code, he'd comment the allocation nobody had noticed, free up a megabyte, and be the hero that made the game fit.
  - btilly 10y ago
    That one is famous. But in the version that I heard, the lead programmer didn't do it to be a hero. He did that because he knew that it is easier to not waste memory up front than it is to figure out where you can save it at the end. Therefore he made sure that memory became an obvious problem early on so that people were careful about it the rest of the way through development. In other words because he did that, the game hit memory targets that it never could have had people waited until they tried to squeeze everything in at the last moment and only then found that they were out of memory.
marknadal 10y ago
Oh boy, it becomes far worse when you are dealing with javascript and web apps. Pretty much all modern day "best practices" are not, but bullet hole leaks of performance. I spent about a year researching this and put together my findings into this (too short) 20 minute tech talk: https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI hopefully others will find it useful too. Or just never use JS ;) but it is possible to make it fast.
songlinhai 10y ago
I did several research projects on performance bugs during my phd. https://songlh.github.io/ https://songlh.github.io/
glangdale 10y ago
We run into this a fair bit of the time with Hyperscan. One particularly noxious anti-pattern that can arise is the "prefilter that isn't" anti-pattern: specifically, a multiple stage pipeline where the initial stage is meant to filter out some portion of false positives when feeding to a more expensive check, but isn't doing its job. The painful thing is that the expensive check downstream winds up doing everything, but the code still "works" because the expensive check is the only necessary part (well, as long as the pre-filter isn't totally hosed and isn't producing false negatives).
glangdale 10y ago
Occasionally I like the idea of producing some sort of dynamic instrumentation code (I privately think of it as "shitgrind") that detects dynamic code that is doing redundant work and flags it. Of course, this doesn't tell you that the code is useless (i.e. I might have to initialize something that subsequently isn't used down some deep and not-anticipatable-at-compile-time set of branches), but it might provide hints that I'm doing a lot of writes that never match up with reads, or that the writes/calculations that I'm doing are going into places that already have that value. It would be incredibly slow but there are plenty of uses I can imagine where you could detect performance problems in a toy version of the problem (maybe running 100-1000x slower) that are the same problems that you would have in the real version of the problem at full scale. Would be a fun project, IMO.
- AstralStorm 10y ago
  Assembly instruction level profiler? Sounds like an extension to callgrind.
  - glangdale 10y ago
    Well, not just instruction profiling, but profiling of effects and tracing of all reads/writes. Very, very expensive.
makecheck 10y ago
Performance sort of has a paradoxical "coffee test": something that takes a few seconds is aggravating and you will wait for it but something that routinely takes 5 minutes will make you go get coffee or switch tasks completely. With enough alternate tasks, waits are absorbed. Oddly, overall throughput can be worse with sporadic tiny delays that are technically indicative of a fast task. You don't just want fast operations, you want programs that can be smart about gathering unavoidable delays into chunks. This can even pay dividends when consuming resources, as the gathered steps may use less (e.g. laptop wakes dormant hardware just once to perform several grouped tasks, instead of repeatedly over a short time frame).
deleted 10y ago
[deleted]
luckydude 10y ago
"It takes an experienced programmer, with a reasonably accurate mental model of the problem and the correct solution, to know how fast the operation should have been performed, and hence if the program is running slower than it should be." I love this sort of approach to performance. If you know the hardware, the software, and the application, you can predict how fast it should go. That's surprisingly rare, in my experience, sadly. About the time I wrote lmbench I was sitting in a lot of meetings at Sun where people were saying "we should do this" and I could do a mental estimate of how fast it would go (I memorized most of the stuff that lmbench measured so I know how many packets/sec we could do, how many iops, how much memory bandwidth we had, etc). You'd be amazed at the number of times "architects" were proposing to build something that couldn't possibly work. It's pretty systems-y and not that common in programmers, what with today's frameworks and all. But there is still some value in being able to predict how fast something should run. I love it when I see people doing that with the needed knowledge to be accurate, very fun to watch.
bitwize 10y ago
An internet points out the obvious -- that performance matters in computing. Hackernews nods in agreement, then goes back to coding Ruby on Rails applications in an editor implemented in HTML5 and JavaScript that consumes 13% CPU just to make the cursor blink.

129 points by Ono-Sendai 10y ago

Reminds me of https://accidentallyquadratic.tumblr.com/ https://accidentallyquadratic.tumblr.com/

Back in 2001, Joel Spolsky coined "Schlemiel the Painter's algorithm" to illustrate the same class of inefficient programming techniques: https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Painter.27s_algorithm https://en.wikipedia.org/wiki/Joel_Spolsky#Schlemiel_the_Pai... His particular example was with the standard C library function strcat where each new invocation traverses the enlarged string over and over from the beginning, in order to find the terminating NUL character and append the new part.

smnscu 10y ago

Already down. edit: up again now https://webcache.googleusercontent.com/search?q=cache:o4ftZJqeuhsJ:https://www.forwardscattering.org/post/49+&cd=1&hl=en&ct=clnk&gl=ro https://webcache.googleusercontent.com/search?q=cache:o4ftZJ...

Ono-Sendai 10y ago
Oh the irony :) Will try to fix. Edit: Back up, let's see how long it lasts now :)
- yetihehe 10y ago
  Less than 15min ;) Edit: back up after further 10min. Seems like it moves in waves.

deleted 10y ago

[deleted]

pjc50 10y ago

This is one of the few areas where adding more levels of abstraction cannot help you fix the problem. It's remarkably easy for some library function or low level of the program to have O(n) performance that ends up getting called in a loop with the same input or output.

gmfawcett 10y ago
It's true that libraries can cause performance issues, but it's not a given. Using template metaprogramming libraries (like STL and Boost in C++) can result in clean, high-level code which has very good performance. Often the "loops" and other high-level constructs are optimized right out of the program. Also, link-time and whole-program optimization can remedy some performance issues across modules/libraries. [edit: not algorithmic problems, of course, LTO isn't going to rewrite your algorithms for you.]
- tyingq 10y ago
  It does, in some situations, create a visibility issue. Very common in java. Using some higher level library like cxf, for example, for https connectivity to services. Unknown to you, and somewhat difficult to unearth (many layers down the rabbit hole to find the actual socket code, not in the cxf code), is that connection pooling doesn't work well, and every connection spawns more than one underlying object.
- AstralStorm 10y ago
  STL or rather standard C++ library can result in clean code. Boost... rarely, especially the more advanced metaprogramming tricks there are esoteric. Neither typically provides highest performance. (I.e. boost ublas vs Eigen)
hacker_9 10y ago
Depends on the problem? Abstractions can usually make assumptions about the data, and thus be faster.
wtetzner 10y ago
> This is one of the few areas where adding more levels of abstraction cannot help you fix the problem. I'm not sure that's true. It is of course true that adding levels of abstraction often degrades performance, but I don't think it's necessary. In fact, programming languages themselves are abstractions. It's pretty well agreed at this point that a good C compiler will generate faster code than someone writing assembly. It's of course possible to write code that's at least as fast in assembly, but in practice, nobody would do it, because the program would be unmaintainable. You can make abstractions fast, and one of the benefits of doing so is that you can make your abstraction be the easy way of doing things. That means, by picking the easiest solutions, the people on your team are also using an efficient solution.

nateberkopec 10y ago

Personal blog authors - if you insist on not using Medium/similar big box blogging service, just host your blog as flat files on S3. It will never, ever go down when you hit number 1 on HN (as this poor author's site has).

skarap 10y ago
It wouldn't go down with flat files even if you hosted it on an arduino connector to your phone's tethered wifi.
- toast0 10y ago
  How many tls handshakes per second can an arduino manage? Https everywhere isn't cheap.
pat2man 10y ago
You can also throw cloudflare in front of it for free.
mrighele 10y ago
An alternative is to cache request at the web server level so that successive requests will not be more heavy than serving static page. Caching just for a few seconds (some people call it microcaching [1]) will let your blog survive without issues. [1] https://www.nginx.com/blog/benefits-of-microcaching-nginx/ https://www.nginx.com/blog/benefits-of-microcaching-nginx/

maximilianroos 10y ago

Ironic that the site is down...

tcopeland 10y ago

At one point I was working on a utility to detect suboptimal sequence of method calls; canonical example is using "[1,2,3].select {|x| x > 1 }.first" rather than "[1,2,3].detect {|x| x > 1}". These can be performance issues, although the bigger win is in readability and communicating the developer intent. More details and examples here if you're interested: https://thomasleecopeland.com/2014/10/22/finding-suboptimal-api-usage.html https://thomasleecopeland.com/2014/10/22/finding-suboptimal-... I haven't worked on it much recently though because the problems it found weren't that significant. But I like the idea of runtime analysis for finding issues, especially in a dynamically-typed language.

draw_down 10y ago

"Should" is a tricky word.

base698 10y ago

In practice the majority of web app performance have more to do with latency of IO. People seem to forget that 4ms may seem fast to humans, but it's glacial to computers. Add 100 4ms calls to the DB and all the sudden your code is waiting for half a second if it does nothing else.

AstralStorm 10y ago
Bonus points that in a real real-time system (anything with a GUI) you might have only a 1 ms or less to handle an event in a bigger stream, e.g. dragging. This is why so much UI is or should be really asynchronous.

unsoundInput 10y ago

Even though I agree that >>insert method of std::map is roughly 7 times slower than it should be<< is bad, these kind of problems are not too hard to find and solve if they are actually problematic for your software. The most problematic performance issues I've come across were usually bad/premature optimizations that were not (correctly) validated against a simpler implementation as a performance baseline. Things like parallelism (multi-threading, web workers) or caching can absolutely tank performance if not done correctly. Plus they usually tend to make stuff more complex and bug-prone.

rosshemsley 10y ago

One of my favourite examples: A friend once sped up some code 90% by changing a list.size() == 0 to list.empty() yes, std::list::size() can be _linear_ in C++98. http://www.cplusplus.com/reference/list/list/size/ http://www.cplusplus.com/reference/list/list/size/

logophobia 10y ago
std::list is a linked list, so it makes sense that size() is linear. Only if you store the size separately, you get constant behavior (which the c++11 enforces), it's constant. Your friend probably could've sped his code up even more by using a std::vector. Linked lists have very bad processor cache behavior (not allocated in continuous memory), there are not a lot of things a list does better than a vector. Only if you prepend a lot, or insert in the middle of a list, it might have some advantages over a vector, even then, there're are better alternatives.
- sqeaky 10y ago
  I have timed it on several implementations and posted (mostly on on reddit.com/r/cpp), but until you get several thousand items std::vector is just faster than std::list. Of course it varies from machine to machine, for example my 4th gen i7 could insert about 8,000 items before list finally got the same speed as inserting at the front of an std::vector and my AMD 8150 (The bulldozer thing way before ryzen) was about 3,500. I did similar tests comparing linear scans through an std::vector and lookups in a std::map or std::unsorted_map, with similar results, many thousands of items are required before dumb vectors got slow. My default rule is to use a std::array if the size is fixed or an std::vector if the size changes until I hit 1,000 items. Then look at the complexity notation of the operations I am doing and choose what containers to then to benchmark. Then I throw a couple microbenchmarks in the test suite and use whatever that tells me will be faster in my case. Sometimes I even go so far as to use ifdefs and typedefs to change the container for different builds. In practice I use a lot of std::vectors and I sort them a lot. Sorted vectors are very fast in the general case. In the few workloads I have with large numbers of items, even with millions of items, I use std::sort and std::binary_search with std::vector because they can be much faster than other associative containers. You do need to use operator< intelligently and that means either already passing in an overload for the comparator function or making a simple container class that wraps these concepts the way that makes sense for your workload.

dahart 10y ago

> Performance bugs are out there, lurking and unseen Is it a bug if it's unseen? Code that runs slower than it could in theory is literally everywhere. But is it worth paying the cost of the time to fix it, if you aren't experiencing any delays? I love the way Michael Abrash talks about optimizing. He emphasizes and re-emphasizes that optimization is for people not computers, and that if you aren't experiencing a delay, you're done. "In other words, high-performance code should ideally run so fast that any further improvement in the code would be pointless. "Notice that the above definition most emphatically does not say anything about making the software as fast as possible. It also does not say anything about using assembly language, or an optimizing compiler, or, for that matter, a compiler at all. It also doesn’t say anything about how the code was designed and written. What it does say is that high-performance code shouldn’t get in the user’s way—and that’s all." http://www.jagregory.com/abrash-black-book/#the-human-element-of-code-optimization http://www.jagregory.com/abrash-black-book/#the-human-elemen...

vilya 10y ago
I think that was true back when he wrote the words, but less so now. Even if you're not noticing a delay, code that completes quicker is generally code that uses less power and you can still notice the effect on your phones battery life - or on the size of your monthly bill from your data center.
- dahart 10y ago
  Great point. Efficiency comes in different shapes. To respond, I'd say power & cost are human oriented metrics, and I take his point to be use a human oriented metric, for me it's as true now as it ever was.
- datenwolf 10y ago
  Ah darn, I wrote along the same line an hour ago or so but got delayed in hitting the submit button until now. Yes, pretty much this.
pdpi 10y ago
"Unseen" is relative A performance issue that is hard to detect at 10 rps becomes an issue when you're trying to push 1000 rps. A performance issue in a mobile app (or a desktop app on a laptop) might be undetectable from a responsiveness or frame rate point of view, but burn through your battery. If you count memory footprint as a performance issue, thowr don't become apparent until you try to run more things concurrently, and find yourself thrashing when memory should be plentiful.

prohor 10y ago

The fun begins when you need to find performance issues in production, where you cannot really use profiler. Then you need to jump into APM tools. Unfortunately it seems there nothing really for that level for free, but take this for example: https://www.dynatrace.com/blog/code-level-visibility-for-node-js/ https://www.dynatrace.com/blog/code-level-visibility-for-nod...

whyever 10y ago
Can't you just use something like performance counters, like perf for Linux?
- itsderek23 10y ago
  In my experience, it's very difficult to tie profiling data from generic profilers to specific requests, then to the specific lines-of-code triggering the problems. This is important because many performance conditions don't reveal themselves all of the time: for example, it's very common that an issue might only be a problem for your largest customers. The context is really important. Scout has a production-safe profiler for Ruby apps that builds on the wonderful StackProf gem that does this: http://help.apm.scoutapp.com/#scoutprof http://help.apm.scoutapp.com/#scoutprof

amorphid 10y ago

If performance matters, benchmarking the heck out of everything helps quite a bit! I like the style of benchmarks that focus on comparing iterations per second.

kazinator 10y ago

It's only a bug if the stakeholders that surround the program agree that there should be (or already is) in place a performance requirement which is not being met. Opportunities to uncover performance that could be improved and impose requirements against it are out there, lurking unseen. Only problem is you have to convince people to care on a case-by-case basis.

kevindqc 10y ago

> when I was working on the Doom 3 BFG edition release, the exactly predicted off-by-one-frame-of-latency input sampling happened, and very nearly shipped What is this predicted off-by-one-frame-of-latency input sampling ?

partycoder 10y ago

Performance issues come in all flavors... - CPU usage (complexity, cache misses...) - Memory usage (leaked resources like memory, connections and handles) - Excessive I/O (networking, files...) - Concurrency (contention, starvation, deadlocks, livelocks...) There's no one size fits all profiler. I find load testing in combination with profiling very useful to get an idea of how a system performs. Usually I keep injecting load until the system fails then find what failed. There are other approaches, like automated performance testing (writing tests with assertions on performance). e.g: https://scalameter.github.io https://scalameter.github.io

deleted 10y ago

[deleted]

bastijn 10y ago

I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud environments) is a real pain here. The google white paper on Dapper [0] and all its implementations [1][2][3] in one form or another are appearing but usually target the Web environments. That means that for my legacy stack with multiple nodes connected via networks of different kinds I have to profile on multiple systems, aggregate different log types manually, think of time syncs (not everything logs utc..) and so on. To be honest we just reached the point where we try to get a European subsidiary project rolling where we address this problem. The time lost to trace down unexplainable time gaps between first displayed result in app and request are weeks, and in multiple occasions months. Just because we have no easy way to profile our distributed ecosystem. After identifying who is to blame: network, db nodes, processing nodes, application logic, cache (invalidation); we can talk about fixing. For fixing of course it would be great if you could extract a meta model of the application and another of the hardware. Changes to those models and some nice prediction of impact would be the ideal end goal. Or, if we are wishing anyway, some nice DSE on those models to optimize would be even better. [0] https://research.google.com/pubs/pub36356.html https://research.google.com/pubs/pub36356.html [1] http://opentracing.io/ http://opentracing.io/ [2] http://lightstep.com/ http://lightstep.com/ [3] https://aws.amazon.com/xray/ https://aws.amazon.com/xray/

pritianka 10y ago
OpenTracing has libraries for C++ (though it needs work tbh) and Java which can be used for mobile. The idea is to cover your entire stack ...
- bastijn 10y ago
  I know. But our stack is C#, Wcf, websockets with c++ where performance is required and some RestAPI/wpf/html5/typescript/javascript ui/clients. The stack is something grown the past 15 or so years (medical business). The existing options are not mature enough yet for us to be used, or not available yet on our platforms.

ptx 10y ago

One recent example of this is Synapse, the Matrix messaging server. Synapse was generally considered to be inefficient and a huge memory hog, so they have been experimenting with various ways to rewrite it in Go, saying[1] at the end of last year: "Synapse is in a relatively stable state currently ... we’re starting to hit some fundamental limitations of the architecture: ... python’s single-threadedness and memory inefficiency; ...; the fact the app papers over SQL problems by caching everything in RAM (resulting in synapse’s high RAM requirements); ..." But in January[2] it was discovered, during the investigation of another problem, that the vast majority of the memory usage was due to a bug: "It’s also revealed the root cause of why Synapse’s RAM usage is quite so bad – it turns out that it actually idles at around 200MB with default caching, but there’s a particular codepath which causes it to spike temporarily by 1GB or so – and that RAM is then not released back to the OS." I wonder if they would still have started the Go rewrite if it hadn't been for that bug. (It seems the new implementation has a better architechture and is substantially faster, so I guess it's still a good idea though, but maybe not as urgent as before?) [1] https://matrix.org/blog/2016/12/26/the-matrix-holiday-special-2016-edition/ https://matrix.org/blog/2016/12/26/the-matrix-holiday-specia... [2] https://matrix.org/blog/2017/01/06/synapse-0-18-6-is-out-please-upgrade-especially-if-on-0-18-5/ https://matrix.org/blog/2017/01/06/synapse-0-18-6-is-out-ple...

Namrog84 10y ago
This is a great example of why you can't take perf related issues intuitively or at face value. Profiling and in depth analysis is always needed. I wonder if this could have been more easily spotted earlier on if a better analysis was done.
BurningFrog 10y ago
So the analysis of why it was slow was just informed guesswork. One thing I keep learning again and again: You never know what the performance problem is until you've measured it.
Arathorn 10y ago
We would absolutely still be doing the go-rewrite: while fixing that bug improved the memory usage, it didn't help actual performance much. Meanwhile Dendrite (the rewrite) is currently benchmarking 2 orders of magnitude faster than Synapse. Unfortunately Synapse's architecture simply has many design efficiencies :)

wolfgang42 10y ago

Quote from the linked article by John Carmack[1]: > The fly-by-wire flight software for the Saab Gripen (a lightweight fighter) went a step further... Control flow went forward only. Sometimes one piece of code had to leave a note for a later piece telling it what to do, but this worked out well for testing: all data was allocated statically, and monitoring those variables gave a clear picture of most everything the software was doing.... No bug has ever been found in the “released for flight” versions of that code. When I started writing Arduino code, I independently re-invented this technique. I write my code with only fixed-size loops and no delay() calls (basically cooperative multitasking), making heavy use of finite-state machines to keep track of what should happen when. This style of code takes some getting used to, but its advantages are enormous: I see other people using delays and busy-loops and then struggling to get interrupts working to keep their device responsive, whereas I have a clean linear set of logic that can be easily understood and is free of strange race conditions and the like, and is guaranteed to keep responding to events. This style also makes it easy to literally eyeball performance: attach a bicolor LED (one that has e.g. red and green LEDs in the same package) and tell it to toggle colors once per loop. If the main loop is taking more than 8 milliseconds or so the LED will start to flicker visibly; if something pauses the LED will go solid and it's obvious the main loop is frozen. [1] http://number-none.com/blow/blog/programming/2014/09/26/carmack-on-inlined-code.html http://number-none.com/blow/blog/programming/2014/09/26/carm... (Edits for clarity and writing style.)

dahart 10y ago
One of the interesting side effects of this style is that you sometimes choose to let code run slower than it could, and choose to use more memory than you need. The point is (as you say) to make all loops & allocation as predictable as possible, so when you make them fixed size, you have to fix it on the worst case. Ideally there is no worst case, and the amount of work you do per frame or cycle is constant. But for anything non-trivial, chances are fixed allocations & loops aren't always needed, even though they're extremely useful. This does exactly what you say, in my experience: it makes performance eyeball-able, you can see where problems are. Code becomes so easy to reason about that debugging is easier and time is saved.
RangerScience 10y ago

ericb 10y ago

I'm open sourcing a Java Selenium library with page performance assertions built in as part of my startup. It will let you do things like: test.config.setSLAMaxTime(2000); // set time in MS assertSLAMet(); // add this to any page It will also allow you to assert specific URL's, and window timings. You can then use these assertions to pass/fail builds. I also have some other nifty ideas on preventing performance regressions I'm building into this (smart benchmarking vs. previous). If anyone is interested, I'll be announcing it here when we open source it: http://signup.browserup.com/ http://signup.browserup.com/

orf 10y ago
> What if your Selenium tests had performance asserts for CI/CD Isn't this just... a single function? Like, how the heck do you build a startup around that. We do this as part of our Python selenium tests, I didn't realize it wasn't built in to be honest.
- ericb 10y ago
  There's a lot more to it--this is just a tiny part we're giving away, not something we'll make money on. We do real-browser load testing in the cloud using the same scripts folks use for integration testing. Because they can also run locally, you can run them alongside your integration tests with each commit, and you always know you have working scripts. Scripts are easy to code and maintain because they can use page objects like functional tests. This means you can have versioned load test/monitoring scripts in your project that always work with a particular sha. When you deploy, you can start transactional monitoring immediately. It also means load testing is not a separate stage in your development lifecycle.

btilly 10y ago

I have seen too many of these to keep track of. But I will never forget once having to work with some Windows software written by someone else. He decided to parse a CSV file by reading the whole thing in memory, and then literally chopping off one character at a time..copying the rest of the string. It made what should be an instantaneous import of a fairly small file into a 20 minute ordeal. Simply reading in one line at a time would have been a massive speedup. I did not have access to change it. I asked him to fix it. I begged him to fix it. His response was to shrug and say, "Who cares? This runs as part of a batch process with nobody in front of it. It works fine." That may be, but I cared because I had to sit through it repeatedly while debugging certain misbehavior that arose in said batch process. Years later I was talking to an ex-coworker who stayed at said company after I left. He reported that that original developer after I left had to make a fix to said batch process. While doing so he had fixed the performance bug and was an instant hero among a number of other people who had to sit through the same import repeatedly over time. Yeah..it turns out that he had made the fix I'd originally asked. And nobody realized that he had caused the problem originally AND refused to fix it. That's one developer that I'm glad I will never have to work with again...

dahart 10y ago
That would be pretty frustrating and annoying! This reminds me of a story I heard about a lead game programmer who made a habit of pre-allocating an unused megabyte at the beginning of a project. The last week before ship, when the game would crash out of memory often and everyone was freaking out and trying to squeeze every last byte out of their code, he'd comment the allocation nobody had noticed, free up a megabyte, and be the hero that made the game fit.
- btilly 10y ago
  That one is famous. But in the version that I heard, the lead programmer didn't do it to be a hero. He did that because he knew that it is easier to not waste memory up front than it is to figure out where you can save it at the end. Therefore he made sure that memory became an obvious problem early on so that people were careful about it the rest of the way through development. In other words because he did that, the game hit memory targets that it never could have had people waited until they tried to squeeze everything in at the last moment and only then found that they were out of memory.

marknadal 10y ago

Oh boy, it becomes far worse when you are dealing with javascript and web apps. Pretty much all modern day "best practices" are not, but bullet hole leaks of performance. I spent about a year researching this and put together my findings into this (too short) 20 minute tech talk: https://youtu.be/BEqH-oZ4UXI https://youtu.be/BEqH-oZ4UXI hopefully others will find it useful too. Or just never use JS ;) but it is possible to make it fast.

songlinhai 10y ago

I did several research projects on performance bugs during my phd. https://songlh.github.io/ https://songlh.github.io/

glangdale 10y ago

We run into this a fair bit of the time with Hyperscan. One particularly noxious anti-pattern that can arise is the "prefilter that isn't" anti-pattern: specifically, a multiple stage pipeline where the initial stage is meant to filter out some portion of false positives when feeding to a more expensive check, but isn't doing its job. The painful thing is that the expensive check downstream winds up doing everything, but the code still "works" because the expensive check is the only necessary part (well, as long as the pre-filter isn't totally hosed and isn't producing false negatives).

glangdale 10y ago

Occasionally I like the idea of producing some sort of dynamic instrumentation code (I privately think of it as "shitgrind") that detects dynamic code that is doing redundant work and flags it. Of course, this doesn't tell you that the code is useless (i.e. I might have to initialize something that subsequently isn't used down some deep and not-anticipatable-at-compile-time set of branches), but it might provide hints that I'm doing a lot of writes that never match up with reads, or that the writes/calculations that I'm doing are going into places that already have that value. It would be incredibly slow but there are plenty of uses I can imagine where you could detect performance problems in a toy version of the problem (maybe running 100-1000x slower) that are the same problems that you would have in the real version of the problem at full scale. Would be a fun project, IMO.

AstralStorm 10y ago
Assembly instruction level profiler? Sounds like an extension to callgrind.
- glangdale 10y ago
  Well, not just instruction profiling, but profiling of effects and tracing of all reads/writes. Very, very expensive.

makecheck 10y ago

Performance sort of has a paradoxical "coffee test": something that takes a few seconds is aggravating and you will wait for it but something that routinely takes 5 minutes will make you go get coffee or switch tasks completely. With enough alternate tasks, waits are absorbed. Oddly, overall throughput can be worse with sporadic tiny delays that are technically indicative of a fast task. You don't just want fast operations, you want programs that can be smart about gathering unavoidable delays into chunks. This can even pay dividends when consuming resources, as the gathered steps may use less (e.g. laptop wakes dormant hardware just once to perform several grouped tasks, instead of repeatedly over a short time frame).

deleted 10y ago

[deleted]

luckydude 10y ago

"It takes an experienced programmer, with a reasonably accurate mental model of the problem and the correct solution, to know how fast the operation should have been performed, and hence if the program is running slower than it should be." I love this sort of approach to performance. If you know the hardware, the software, and the application, you can predict how fast it should go. That's surprisingly rare, in my experience, sadly. About the time I wrote lmbench I was sitting in a lot of meetings at Sun where people were saying "we should do this" and I could do a mental estimate of how fast it would go (I memorized most of the stuff that lmbench measured so I know how many packets/sec we could do, how many iops, how much memory bandwidth we had, etc). You'd be amazed at the number of times "architects" were proposing to build something that couldn't possibly work. It's pretty systems-y and not that common in programmers, what with today's frameworks and all. But there is still some value in being able to predict how fast something should run. I love it when I see people doing that with the needed knowledge to be accurate, very fun to watch.

bitwize 10y ago

An internet points out the obvious -- that performance matters in computing. Hackernews nods in agreement, then goes back to coding Ruby on Rails applications in an editor implemented in HTML5 and JavaScript that consumes 13% CPU just to make the cursor blink.