4 ms·

Jeff Atwood is wrong about performance

62 points by vaksel 17y ago

diN0bot 17y ago
excellent points. plus the author's writing style is direct and if not humble, than certainly not arrogant.
patio11 17y ago
If you have upgraded all your servers without consolidating, year after year, you will notice that your electricity bill is going through the roof. I don't know, you can buy an awful lot of electricity for an hour of my time, and I'm cheap... I think the largest deployment my day job supports probably causes an electricity bill in the hundreds a year range. If you had a meeting between one PL and two engineers to discuss what to do about the electricity bill you could cancel the meeting and pre-pay the next few years. Adjust if you're Google... but you're not Google.
- ankp 17y ago
  It's not cheap, and you don't have to be Google for it to matter. Most of the time, the costs are just non-obvious to engineering. Here are some rough numbers. Power for a single 48U rack usually runs $200-$300/month for two 20amp circuits (possibly +2 for redundancy). A modern 1U server draws anywhere from (roughly) 1.66 to 2.5 amps, and you can only use 80% of a circuit's maximum supported load due to safety regulations, resulting in a total supported count of between 12 to 19 mid-range servers per rack. Higher-end servers have a significantly increased power draw, but potentially higher efficiency, and there can be a significant advantage to leveraging multi-core systems. In addition to rackspace and power, each server must obviously be connected to the network. The switches draw enough power to potentially decrease the number of supported servers. Each rack costs roughly $500/month (plus power), and often requires advanced provisioning to ensure that rackspace is available, and if possible, located in the same area. It's not unusual to buy an entire block of space well ahead of actual use to ensure that the business' growth can be accommodated. Additionally, one requires an operations staff to maintain the servers. Hard drives fail, backups must be done, hardware must be ordered, tested, and finally, deployed. Simple deployment of a single server, including the networking configuration, travel time, and physical work can easily consume 4-8 hours of operations staff time. When you add together the operational and capital expenditures, I'm not really convinced that the idea is so simple as "servers are cheap, programmers are expensive". Servers, electricity, physical space, and the labor necessary to install and maintain them are not particularly cheap. I'd propose a new mantra: programmers are expensive, and so are servers and IT, and so let's try to not be monomaniacally flippant about it. Software can be easy to write AND perform reasonably well, and to say otherwise simply appears to be a false dichotomy fronted by those who benefit from it most.
  - j_baker 17y ago
    You're still talking about a minimum of 12-19 servers though. I'd imagine that the average website doesn't use even that. I think that SO in particular is at 3 servers. Maybe if SO explodes, this will become an issue, but an optimization would likely have to make a difference of a few racks before it it's even worth considering making an optimization based soley on power concerns.
    - ankp 17y ago
      Whether an optimization is worth pursuing depends on the business requirements, how much the it costs to implement, and whether there are any additional gains beyond avoiding power, space, and personnel costs. Simply picking an efficient (both for the developer, and in terms of implementation) language/platform can be enough to start with, and will help you avoid being boxed in later.
  - lightcrest 17y ago
    You forgot to mention bandwidth as part of the fundamental datacenter cost structure. Some datacenter operators are charging up to $200/month to drop dark fiber into your cage/rack/cabinet. It is then up to you to negotiate a primary and secondary connectivity provider. At non-bulk rates (meaning around 10-100Mbps) this can be up to $50/megabit. Bulk rates look better - but only because if you're pushing that much traffic, you must be rich. This is part of the reason cloud computing should be taken with a grain of salt. There can be fantastic economies of scale when applied correctly (say, on a properly built CDN solution) - but be wary whenever "cheap" and "computing" are mentioned in the same sentence. Someone has to be paying for all of that power, space, and bandwidth.
edw519 17y ago
"Hardware is cheap, programmers are expensive." Mediocre programmers are expensive. Good programmers are the bargain of the century. If companies would just wise up enough to pay a good programmer 3 times as much as a mediocre programmer to do 10 times the work, do it right, and do it so that it can be maintained reasonably, Jeff Atwood's tradeoff would become moot. But companies generally don't do this, which is probably one of the main reasons the best programmers go off and do their own.
- vaksel 17y ago
  what separates a good programmer from a mediocre one?
  - thras 17y ago
    The same thing that separates a good plumber from a mediocre plumber.
    - vaksel 17y ago
      and that would be?
      - deleted 17y ago
        [deleted]
      - wglb 17y ago
        Pipes/abstractions don't leak?
  - edw519 17y ago
    Great question. Ask it to n programmers and get n^2 responses. This could easily be the subject for another post or even a book. Just off the top of my head in no particular order: - understands the problem at hand before writing any code - uses the right tool for the right job - follows accepted standards and protocols without sacrificing creativity - names variables & functions what they actually are for the next programmer - anticipates what could go wrong before relying on a debugger or testing - understands the underlying architecture and how to best utilitze it - never writes the same code twice - never writes in 150 lines that which could be written in 100 lines - Poor code: uncommented. Mediocre: commented. Good: doesn't need comments. - understands the entire code life cycle & writes it to last - has pity on the poor soul who has to maintain it & leaves a clue or 2 - writes flexibly enough to be easily changed before the project is done I could go on and on, but you get the idea. In general... A good programmer writes it right, once, in a week. A mediocre programmer writes it OK, in 2 months, and then futzes with it forever. A bad programmer never gets it done.
antirez 17y ago
I guess that when the problem is to really scale a lot the big cost may be the energy not the hardware, so the performance the coder can bring you for watt is crucial. And to use few CPU cycles you need a programmer that is smart enough to know both computer science to select the best algorithms and data structures and low level programming languages to lower the constant times involved in this algorithms.
wglb 17y ago
One item left out is floor space. This is a serious consideration, maybe more serious than electricity. Real estate for your server racks has a cost associated with it, and it is not so easy to buy/rent in rack-sized increments.
al3x 17y ago
This is good advice. ankp's comment is also good advice. A well-run technical organization shouldn't have to choose between good programmers, code that performs, and the ability to quickly operationalize new hardware when necessary. If you're having to choose, you've got more work to do.
sho 17y ago
Nice straw man. Did anyone ever try to say that it's not worth fixing a 1000x-slowdown error because "we'll just buy 1000 times the hardware, that's cheaper!"? Obviously there is a balance somewhere between pointless, unmaintainable, expensive over-optimisation on one end and programmer incompetence, bad tool choices and ignorance of hardware limitations on the other. There will be a sweet spot in the middle, possibly different for all projects, which just has to be figured out by experienced people who knows what they're doing. Why all this trying to take "sides"? It's like getting all fired up about which is "better", planes or ships. They are both good for different purposes, choose the one that fits your need!
jhawk28 17y ago
Its a short term vs long term analysis. If you focus on the long term, programmers are cheaper. Short term, it is easier to throw more hardware at the problem. The part that makes it difficult to measure is estimating how fast it will run on the new hardware. Usually if you use good algorithms and data structures with an architecture that matches the problem, adding hardware will scale the performance. Mediocre programmers rarely can get this right unless they are lucky.
korch 17y ago
"Jeff Atwood is wrong on the Internet", film at 11. Why is this news?

62 points by vaksel 17y ago

excellent points. plus the author's writing style is direct and if not humble, than certainly not arrogant.

If you have upgraded all your servers without consolidating, year after year, you will notice that your electricity bill is going through the roof. I don't know, you can buy an awful lot of electricity for an hour of my time, and I'm cheap... I think the largest deployment my day job supports probably causes an electricity bill in the hundreds a year range. If you had a meeting between one PL and two engineers to discuss what to do about the electricity bill you could cancel the meeting and pre-pay the next few years. Adjust if you're Google... but you're not Google.

ankp 17y ago
It's not cheap, and you don't have to be Google for it to matter. Most of the time, the costs are just non-obvious to engineering. Here are some rough numbers. Power for a single 48U rack usually runs $200-$300/month for two 20amp circuits (possibly +2 for redundancy). A modern 1U server draws anywhere from (roughly) 1.66 to 2.5 amps, and you can only use 80% of a circuit's maximum supported load due to safety regulations, resulting in a total supported count of between 12 to 19 mid-range servers per rack. Higher-end servers have a significantly increased power draw, but potentially higher efficiency, and there can be a significant advantage to leveraging multi-core systems. In addition to rackspace and power, each server must obviously be connected to the network. The switches draw enough power to potentially decrease the number of supported servers. Each rack costs roughly $500/month (plus power), and often requires advanced provisioning to ensure that rackspace is available, and if possible, located in the same area. It's not unusual to buy an entire block of space well ahead of actual use to ensure that the business' growth can be accommodated. Additionally, one requires an operations staff to maintain the servers. Hard drives fail, backups must be done, hardware must be ordered, tested, and finally, deployed. Simple deployment of a single server, including the networking configuration, travel time, and physical work can easily consume 4-8 hours of operations staff time. When you add together the operational and capital expenditures, I'm not really convinced that the idea is so simple as "servers are cheap, programmers are expensive". Servers, electricity, physical space, and the labor necessary to install and maintain them are not particularly cheap. I'd propose a new mantra: programmers are expensive, and so are servers and IT, and so let's try to not be monomaniacally flippant about it. Software can be easy to write AND perform reasonably well, and to say otherwise simply appears to be a false dichotomy fronted by those who benefit from it most.
- j_baker 17y ago
  You're still talking about a minimum of 12-19 servers though. I'd imagine that the average website doesn't use even that. I think that SO in particular is at 3 servers. Maybe if SO explodes, this will become an issue, but an optimization would likely have to make a difference of a few racks before it it's even worth considering making an optimization based soley on power concerns.
  - ankp 17y ago
    Whether an optimization is worth pursuing depends on the business requirements, how much the it costs to implement, and whether there are any additional gains beyond avoiding power, space, and personnel costs. Simply picking an efficient (both for the developer, and in terms of implementation) language/platform can be enough to start with, and will help you avoid being boxed in later.
- lightcrest 17y ago
  You forgot to mention bandwidth as part of the fundamental datacenter cost structure. Some datacenter operators are charging up to $200/month to drop dark fiber into your cage/rack/cabinet. It is then up to you to negotiate a primary and secondary connectivity provider. At non-bulk rates (meaning around 10-100Mbps) this can be up to $50/megabit. Bulk rates look better - but only because if you're pushing that much traffic, you must be rich. This is part of the reason cloud computing should be taken with a grain of salt. There can be fantastic economies of scale when applied correctly (say, on a properly built CDN solution) - but be wary whenever "cheap" and "computing" are mentioned in the same sentence. Someone has to be paying for all of that power, space, and bandwidth.

edw519 17y ago

"Hardware is cheap, programmers are expensive." Mediocre programmers are expensive. Good programmers are the bargain of the century. If companies would just wise up enough to pay a good programmer 3 times as much as a mediocre programmer to do 10 times the work, do it right, and do it so that it can be maintained reasonably, Jeff Atwood's tradeoff would become moot. But companies generally don't do this, which is probably one of the main reasons the best programmers go off and do their own.

vaksel 17y ago
what separates a good programmer from a mediocre one?
- thras 17y ago
  The same thing that separates a good plumber from a mediocre plumber.
  - vaksel 17y ago
    and that would be?
    - deleted 17y ago
      [deleted]
    - wglb 17y ago
      Pipes/abstractions don't leak?
- edw519 17y ago
  Great question. Ask it to n programmers and get n^2 responses. This could easily be the subject for another post or even a book. Just off the top of my head in no particular order: - understands the problem at hand before writing any code - uses the right tool for the right job - follows accepted standards and protocols without sacrificing creativity - names variables & functions what they actually are for the next programmer - anticipates what could go wrong before relying on a debugger or testing - understands the underlying architecture and how to best utilitze it - never writes the same code twice - never writes in 150 lines that which could be written in 100 lines - Poor code: uncommented. Mediocre: commented. Good: doesn't need comments. - understands the entire code life cycle & writes it to last - has pity on the poor soul who has to maintain it & leaves a clue or 2 - writes flexibly enough to be easily changed before the project is done I could go on and on, but you get the idea. In general... A good programmer writes it right, once, in a week. A mediocre programmer writes it OK, in 2 months, and then futzes with it forever. A bad programmer never gets it done.

antirez 17y ago

I guess that when the problem is to really scale a lot the big cost may be the energy not the hardware, so the performance the coder can bring you for watt is crucial. And to use few CPU cycles you need a programmer that is smart enough to know both computer science to select the best algorithms and data structures and low level programming languages to lower the constant times involved in this algorithms.

wglb 17y ago

One item left out is floor space. This is a serious consideration, maybe more serious than electricity. Real estate for your server racks has a cost associated with it, and it is not so easy to buy/rent in rack-sized increments.

al3x 17y ago

This is good advice. ankp's comment is also good advice. A well-run technical organization shouldn't have to choose between good programmers, code that performs, and the ability to quickly operationalize new hardware when necessary. If you're having to choose, you've got more work to do.

sho 17y ago

Nice straw man. Did anyone ever try to say that it's not worth fixing a 1000x-slowdown error because "we'll just buy 1000 times the hardware, that's cheaper!"? Obviously there is a balance somewhere between pointless, unmaintainable, expensive over-optimisation on one end and programmer incompetence, bad tool choices and ignorance of hardware limitations on the other. There will be a sweet spot in the middle, possibly different for all projects, which just has to be figured out by experienced people who knows what they're doing. Why all this trying to take "sides"? It's like getting all fired up about which is "better", planes or ships. They are both good for different purposes, choose the one that fits your need!

jhawk28 17y ago

Its a short term vs long term analysis. If you focus on the long term, programmers are cheaper. Short term, it is easier to throw more hardware at the problem. The part that makes it difficult to measure is estimating how fast it will run on the new hardware. Usually if you use good algorithms and data structures with an architecture that matches the problem, adding hardware will scale the performance. Mediocre programmers rarely can get this right unless they are lucky.

korch 17y ago

"Jeff Atwood is wrong on the Internet", film at 11. Why is this news?