5 ms·

The author achieved a big speed increase with a faster disk, more RAM, adding indexes, and rewriting slow queries. The relevant skillset is "knowing a thing or

The author achieved a big speed increase with a faster disk, more RAM, adding indexes, and rewriting slow queries. The relevant skillset is "knowing a thing or two about databases", not "full-stack programmer".

fleitz 16y ago
Incorrect, the point about being a full stack programmer is knowing where to look. You only need to know about databases to FIX the problem but you need the full stack to FIND the problem. If you knew a little about assembler then maybe you'd waste a lot of time rewriting code. Instead of fixing the db. Full stack programmers are valuable for their knowledge that is an inch deep but a mile wide. Any fix a full stack performs is going to seem trivial to an expert in that field.
- protomyth 16y ago
  A "full stack programmer" will be able to look at the whole and see what should be done differently, but a database performance person (not a generic DBA) will be bring a much deeper knowledge of that one area (just as a GPU programmer will know more about GPU optimization then the application developer). Sometimes you need that specialist to get you over the hump. An "inch deep and mile wide" sort of knowledge will not get you the fastest performance.
  - dorianj 16y ago
    It won't get you the fastest performance, but usually it will get you fast enough performance. Knowing enough to not do bone-headed things is what gets you acceptable performance. And, it gives you the ability to know where your critical paths are, and to hand those critical paths off to the most expert person in the room.
  - pmiller2 16y ago
    It's been my experience that most of the time, you don't need "the fastest performance." You really only need things to be "fast enough." Though the dividing line between "too slow" and "fast enough" is pretty murky, it's obvious when you're firmly on one side or the other. In the article example, 24 hours is clearly too slow, while 10 minutes is clearly fast enough. Consider, too, that expert-level knowledge of the sort a DB performance person will bring is often (rightly or wrongly) more expensive than that of a generalist. Maybe the expert can bring the runtimes down by a further factor of 100 (to 6 seconds), but, at that point, you have to seriously question whether it's worth it. They might be able to get another factor of 100 and get the runtime for a single analysis down to 6 seconds, but at that point, you have to start seriously questioning whether it would be worthwhile to do it. In the article's example, the analysis step could only be run twice in a week because it took 24 hours to execute (and, then, presumably the other 3 days of the week were spent doing something with the results). Since it might take a full day and a half of work to actually do anything with the results, getting the time to run an analysis down from 10 minutes to 6 seconds clearly isn't worth it. If it were me, I might have even stopped once I got the runtime down below an hour, because then it could be run over a lunch break or during a meeting or something. I suppose my point is basically equivalent to Knuth's quip about premature optimization. In this particular example, there seems to be no reason to optimize further, because the system is probably 95% as useful as it can possibly get. Eliminating 47.8 hours of runtime per week lets you squeeze in one more run per week, but cutting that runtime down to 12 seconds per week (which is essentially 0) would seem to have a very small marginal gain.
    - j_baker 16y ago
      Wait a minute, who says that the specialist also can't do "fast enough"? And more to the point, how do you know that the "fast enough" solution is within the reach of the generalist or that the specialist can't figure it out sooner than the generalist can? I can't count the number of times I've spun my wheels on something for days and then brought the problem to an experienced database developer who figured it out in minutes.
      - khafra 16y ago
        I think his point is that there's usually a bottleneck somewhere, and unless the bottleneck lies within the specialist's realm of specialization, the generalist is in a better position to spot it.
        j_baker 16y ago
        What are you trying say? That there are pros and cons behind the decision between specialization and generalization? That's crazy talk if ever I heard it.

fleitz 16y ago

Incorrect, the point about being a full stack programmer is knowing where to look. You only need to know about databases to FIX the problem but you need the full stack to FIND the problem. If you knew a little about assembler then maybe you'd waste a lot of time rewriting code. Instead of fixing the db. Full stack programmers are valuable for their knowledge that is an inch deep but a mile wide. Any fix a full stack performs is going to seem trivial to an expert in that field.

protomyth 16y ago
A "full stack programmer" will be able to look at the whole and see what should be done differently, but a database performance person (not a generic DBA) will be bring a much deeper knowledge of that one area (just as a GPU programmer will know more about GPU optimization then the application developer). Sometimes you need that specialist to get you over the hump. An "inch deep and mile wide" sort of knowledge will not get you the fastest performance.
- dorianj 16y ago
  It won't get you the fastest performance, but usually it will get you fast enough performance. Knowing enough to not do bone-headed things is what gets you acceptable performance. And, it gives you the ability to know where your critical paths are, and to hand those critical paths off to the most expert person in the room.
- pmiller2 16y ago
  It's been my experience that most of the time, you don't need "the fastest performance." You really only need things to be "fast enough." Though the dividing line between "too slow" and "fast enough" is pretty murky, it's obvious when you're firmly on one side or the other. In the article example, 24 hours is clearly too slow, while 10 minutes is clearly fast enough. Consider, too, that expert-level knowledge of the sort a DB performance person will bring is often (rightly or wrongly) more expensive than that of a generalist. Maybe the expert can bring the runtimes down by a further factor of 100 (to 6 seconds), but, at that point, you have to seriously question whether it's worth it. They might be able to get another factor of 100 and get the runtime for a single analysis down to 6 seconds, but at that point, you have to start seriously questioning whether it would be worthwhile to do it. In the article's example, the analysis step could only be run twice in a week because it took 24 hours to execute (and, then, presumably the other 3 days of the week were spent doing something with the results). Since it might take a full day and a half of work to actually do anything with the results, getting the time to run an analysis down from 10 minutes to 6 seconds clearly isn't worth it. If it were me, I might have even stopped once I got the runtime down below an hour, because then it could be run over a lunch break or during a meeting or something. I suppose my point is basically equivalent to Knuth's quip about premature optimization. In this particular example, there seems to be no reason to optimize further, because the system is probably 95% as useful as it can possibly get. Eliminating 47.8 hours of runtime per week lets you squeeze in one more run per week, but cutting that runtime down to 12 seconds per week (which is essentially 0) would seem to have a very small marginal gain.
  - j_baker 16y ago
    Wait a minute, who says that the specialist also can't do "fast enough"? And more to the point, how do you know that the "fast enough" solution is within the reach of the generalist or that the specialist can't figure it out sooner than the generalist can? I can't count the number of times I've spun my wheels on something for days and then brought the problem to an experienced database developer who figured it out in minutes.
    - khafra 16y ago
      I think his point is that there's usually a bottleneck somewhere, and unless the bottleneck lies within the specialist's realm of specialization, the generalist is in a better position to spot it.
      - j_baker 16y ago
        What are you trying say? That there are pros and cons behind the decision between specialization and generalization? That's crazy talk if ever I heard it.