5 ms·

> If I can't make your worst-performing page load 50% faster, you don't have to pay the bill. Interesting approach. This is also nicely recursive, with each ru

by majke 11y ago

> If I can't make your worst-performing page load 50% faster, you don't have to pay the bill. Interesting approach. This is also nicely recursive, with each run being more expensive. I would prefer that statement to have a cost cap. Say "... in three days of work".

jstanley 11y ago
Sure, I would work out something mutually agreeable before saying "tadaa! That's 9 grand please!" :) Thanks for reading.
collyw 11y ago
Just buy a bigger box with double the memory?
- Piskvorrr 11y ago
  That's the easy way out - therefore, step 0. Step 1 is "we tried throwing hardware at it, didn't help a lot, now what?" ("Well throw more hardware then" rarely gives a measurable improvement.) To the downvoter: Maybe in your fantasy world, throwing more computing power at problems solves Eeeverything, including P ?= NP. I do wonder - why people even bother offering such services as the OP, if it's so pointless?
  - dr_zoidberg 11y ago
    I think you may have been downvoted because the article just explains: step 0 should always be profiling and then deciding what you need, either more hardware or better code. I once spent weeks optimizing code on a "best in the world" (for my branch of work) open source project because it made the assumption that it'd run on SSDs, hence it made zillions of small reads from the disk. There were two approaches to solve that: buy an SSD, or refactor the code. But it was through profiling that I found the answer (and in my economic situation at the time, buying an SSD was definitely not viable, much less a step 0!).
- jstanley 11y ago
  Unless the box is heavily overloaded, that won't help. Profile it and see what needs fixing!
- jeffasinger 11y ago
  Sometimes that doesn't help. At work we were debugging why a process that is usually quite fast (< 5min) was taking close to 2 hours once we got to the load we predict for this coming fall. We noticed the box it was running on was running out of memory, so we assumed resizing it to have ~4x the RAM would help, but surprisingly it didn't. The actual answer turned out to be algorithmic, and we were able to rewrite it to use an O(n) algorithm rather than an O(n^2) algorithm.
- phkahler 11y ago
  Some companies box is a warehouse with rows of racks of servers consuming hundreds of kilowatts, if not a megawatt. Those are exceptional cases, but it helps to point out how "just buy a bigger box" eventually has limits.
- mootothemax 11y ago
  >Just buy a bigger box with double the memory? For what it's worth, a quite staggering number of sites I've fixed up have had databases with varchar ID columns, and no indexes anywhere. Thankfully the DB sizes have been small enough that fixing this has taken a few seconds to a few minutes max of DB downtime, with instant, seriously-impressive-looking results afterwards.
  - jimcsharp 11y ago
    I wonder when this type of work will go away. I see it everywhere.
    - CaptSpify 11y ago
      When companies figure out that hiring "one guy that can do everything by himself!" is a bad idea. So...... not likely anytime soon.
      - mgkimsal 11y ago
        Except... sometimes one guy really can do everything by himself, and do it much better than the other folks. However, usually the person who can do that is "too expensive" and they'd rather spread the cost (or a higher cost) over 3-4 people. It's not just talent/skills - there's more communication overhead when dealing with a team, and when the team has a range of skill levels on it, the comm overhead can make things worse (if not managed well)
        Joeri 11y ago
        I have seen the one guy replaced by a team of 5. The team got less done than the guy. I have a longstanding suspicion that developer cost climbs slower than developer skill and productivity, meaning companies should always hire the best they can get, because they'll maximize productivity that way.
        sokoloff 11y ago
- pjc50 11y ago
  Hah. Why think when you can spend money? Firstly, this doesn't help if you're CPU-bound or write-bandwidth-bound, both of which are quite common. Secondly, you may not be able to simply get a bigger box. Most motherboards have effective RAM limits. Bigger iron gets exponentially more expensive. Jeff Atwood on this: http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidden-costs/ http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidde... (I remember about a decade ago when our chip design startup bought a 128GB RAM box because that was the largest available at sane pricing ...)
- MrFoof 11y ago
  Fun things I discovered in just my last performance tuning engagement: * Database with homegrown software load-balancer in front of it, and the database replicated to a half-dozen subscribers. Load balancing required database calls, and the subscribers had their LUNs on the same array as the main database. So 7 different servers, but all beating the same storage. In the end, we ended up with no load balancing (just the single instance). * Application server with lots of agents (one on every workstation). Agents were checking in every 200ms, even if they had nothing to report. Result was lots of tiny database transactions (for logging no activity). Throttled that down to 10 seconds (product managers agreeing that was the more realistic threshold) and batching the updates, made the logging asynchronous, and saw the database transaction log no longer waiting on log writes constantly. * Bad clustering keys for tables. Some tables that were being searched constantly as heaps. Sharded tables with divergent schemas. This was fun, as had to conform everything while it was live (got agreement on what was reasonable delay for synch jobs). Chose correct clustering keys, compressed the tables, and then got rid of other indexes that were no longer necessary. Cut IO by 99%, and writes were now just appends. * Badly written queries using inadequate indexes on badly structured tables. TL;DR: An analysis process took 24 hours, and was tuned to take 4.5 hours. I spent a day on it to cut it down to an hour while I worked on other things. We then spent 2 weeks refactoring it to cut it down to about 8 minutes, despite the underlying data model being bleh. We also added ad-hoc functionality to do stuff out-of-band of the regular scheduled job, with an average response time of 6ms. * ORM lazy loading by default, including for things that constantly retrieved collections. Rewrote them to be programmatically eager loaded and reduced database calls from millions a day to thousands a day. * And a little bit of hardware. SAN HBA links were moved form dual 1Gbps to dual 10Gbps. Some SSDs added. Ended up reducing IO latency by 96-98%, depending on the server. Needed to happen, as IO latency was in hundreds of ms, and we were able to get it down to where it should be (< 10ms for reads, 1-3ms for writes). Non-scaling SAAS stack went from falling on its face with an entire rack of hardware, to being cut down to 1/3rd of a rack of hardware and the hardware basically twiddling its thumbs. The organization was in the process of building of entirely new SAAS platform in a completely different tech stack -- they immediately started questioning that decision when the old stack handled everything trivially, and they knew there was a lot more tuning that still could be done.
hyperpape 11y ago
Not calling this dishonest, because it's a fine way to give a guarantee to people who'd don't think much about performance. However, it's one of those claims that I suspect sounds more impressive than it is, depending on how you read it. Unless your system is stupidly simple or has had significant attention paid to performance, there's a good chance your worst page is spending most of its time in some awful O(n^2) algorithm, and will be an easy fix.
- pjc50 11y ago
  The world is full of things that easy to fix if you have the experience, but remain unfixed. > "In the 1880s, James MacNeill Whistler, as plaintiff in a libel action, was challenged, "For two days' labour, you ask two hundred guineas?" "No, I ask it for the experience of a lifetime." That seems an apt summary of the message of this legend." http://www.snopes.com/business/genius/where.asp http://www.snopes.com/business/genius/where.asp

by majke 11y ago

jstanley 11y ago

Sure, I would work out something mutually agreeable before saying "tadaa! That's 9 grand please!" :) Thanks for reading.

collyw 11y ago

Just buy a bigger box with double the memory?

Piskvorrr 11y ago
That's the easy way out - therefore, step 0. Step 1 is "we tried throwing hardware at it, didn't help a lot, now what?" ("Well throw more hardware then" rarely gives a measurable improvement.) To the downvoter: Maybe in your fantasy world, throwing more computing power at problems solves Eeeverything, including P ?= NP. I do wonder - why people even bother offering such services as the OP, if it's so pointless?
- dr_zoidberg 11y ago
  I think you may have been downvoted because the article just explains: step 0 should always be profiling and then deciding what you need, either more hardware or better code. I once spent weeks optimizing code on a "best in the world" (for my branch of work) open source project because it made the assumption that it'd run on SSDs, hence it made zillions of small reads from the disk. There were two approaches to solve that: buy an SSD, or refactor the code. But it was through profiling that I found the answer (and in my economic situation at the time, buying an SSD was definitely not viable, much less a step 0!).
jstanley 11y ago
Unless the box is heavily overloaded, that won't help. Profile it and see what needs fixing!
jeffasinger 11y ago
Sometimes that doesn't help. At work we were debugging why a process that is usually quite fast (< 5min) was taking close to 2 hours once we got to the load we predict for this coming fall. We noticed the box it was running on was running out of memory, so we assumed resizing it to have ~4x the RAM would help, but surprisingly it didn't. The actual answer turned out to be algorithmic, and we were able to rewrite it to use an O(n) algorithm rather than an O(n^2) algorithm.
phkahler 11y ago
Some companies box is a warehouse with rows of racks of servers consuming hundreds of kilowatts, if not a megawatt. Those are exceptional cases, but it helps to point out how "just buy a bigger box" eventually has limits.
mootothemax 11y ago
>Just buy a bigger box with double the memory? For what it's worth, a quite staggering number of sites I've fixed up have had databases with varchar ID columns, and no indexes anywhere. Thankfully the DB sizes have been small enough that fixing this has taken a few seconds to a few minutes max of DB downtime, with instant, seriously-impressive-looking results afterwards.
- jimcsharp 11y ago
  I wonder when this type of work will go away. I see it everywhere.
  - CaptSpify 11y ago
    When companies figure out that hiring "one guy that can do everything by himself!" is a bad idea. So...... not likely anytime soon.
    - mgkimsal 11y ago
      Except... sometimes one guy really can do everything by himself, and do it much better than the other folks. However, usually the person who can do that is "too expensive" and they'd rather spread the cost (or a higher cost) over 3-4 people. It's not just talent/skills - there's more communication overhead when dealing with a team, and when the team has a range of skill levels on it, the comm overhead can make things worse (if not managed well)
      - Joeri 11y ago
        I have seen the one guy replaced by a team of 5. The team got less done than the guy. I have a longstanding suspicion that developer cost climbs slower than developer skill and productivity, meaning companies should always hire the best they can get, because they'll maximize productivity that way.
        sokoloff 11y ago
pjc50 11y ago
Hah. Why think when you can spend money? Firstly, this doesn't help if you're CPU-bound or write-bandwidth-bound, both of which are quite common. Secondly, you may not be able to simply get a bigger box. Most motherboards have effective RAM limits. Bigger iron gets exponentially more expensive. Jeff Atwood on this: http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidden-costs/ http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidde... (I remember about a decade ago when our chip design startup bought a 128GB RAM box because that was the largest available at sane pricing ...)
MrFoof 11y ago
Fun things I discovered in just my last performance tuning engagement: * Database with homegrown software load-balancer in front of it, and the database replicated to a half-dozen subscribers. Load balancing required database calls, and the subscribers had their LUNs on the same array as the main database. So 7 different servers, but all beating the same storage. In the end, we ended up with no load balancing (just the single instance). * Application server with lots of agents (one on every workstation). Agents were checking in every 200ms, even if they had nothing to report. Result was lots of tiny database transactions (for logging no activity). Throttled that down to 10 seconds (product managers agreeing that was the more realistic threshold) and batching the updates, made the logging asynchronous, and saw the database transaction log no longer waiting on log writes constantly. * Bad clustering keys for tables. Some tables that were being searched constantly as heaps. Sharded tables with divergent schemas. This was fun, as had to conform everything while it was live (got agreement on what was reasonable delay for synch jobs). Chose correct clustering keys, compressed the tables, and then got rid of other indexes that were no longer necessary. Cut IO by 99%, and writes were now just appends. * Badly written queries using inadequate indexes on badly structured tables. TL;DR: An analysis process took 24 hours, and was tuned to take 4.5 hours. I spent a day on it to cut it down to an hour while I worked on other things. We then spent 2 weeks refactoring it to cut it down to about 8 minutes, despite the underlying data model being bleh. We also added ad-hoc functionality to do stuff out-of-band of the regular scheduled job, with an average response time of 6ms. * ORM lazy loading by default, including for things that constantly retrieved collections. Rewrote them to be programmatically eager loaded and reduced database calls from millions a day to thousands a day. * And a little bit of hardware. SAN HBA links were moved form dual 1Gbps to dual 10Gbps. Some SSDs added. Ended up reducing IO latency by 96-98%, depending on the server. Needed to happen, as IO latency was in hundreds of ms, and we were able to get it down to where it should be (< 10ms for reads, 1-3ms for writes). Non-scaling SAAS stack went from falling on its face with an entire rack of hardware, to being cut down to 1/3rd of a rack of hardware and the hardware basically twiddling its thumbs. The organization was in the process of building of entirely new SAAS platform in a completely different tech stack -- they immediately started questioning that decision when the old stack handled everything trivially, and they knew there was a lot more tuning that still could be done.

hyperpape 11y ago

Not calling this dishonest, because it's a fine way to give a guarantee to people who'd don't think much about performance. However, it's one of those claims that I suspect sounds more impressive than it is, depending on how you read it. Unless your system is stupidly simple or has had significant attention paid to performance, there's a good chance your worst page is spending most of its time in some awful O(n^2) algorithm, and will be an easy fix.

pjc50 11y ago
The world is full of things that easy to fix if you have the experience, but remain unfixed. > "In the 1880s, James MacNeill Whistler, as plaintiff in a libel action, was challenged, "For two days' labour, you ask two hundred guineas?" "No, I ask it for the experience of a lifetime." That seems an apt summary of the message of this legend." http://www.snopes.com/business/genius/where.asp http://www.snopes.com/business/genius/where.asp