4 ms·

I think we can learn from some other (not-so-obviously-related) disciplines. The game of go[0] has a ranking system where players progress from 30kyu (complete

by hasenj 12y ago

I think we can learn from some other (not-so-obviously-related) disciplines. The game of go[0] has a ranking system where players progress from 30kyu (complete beginner) to 1kyu, then 1dan to 9dan (very strong). [0]: http://en.wikipedia.org/wiki/Go_ranks_and_ratings http://en.wikipedia.org/wiki/Go_ranks_and_ratings It's mostly statistical and based around comparing your skill level to other players. While a 7kyu level is not necessarily that well defined, and might differ between different ranking systems, there's still a meaning to the difference between the levels. When a 5kyu (stronger) is playing a 7kyu, the 7kyu player is given 2 stones in advance (handicap) to make the game even. The number of stones given as a handicap is determined by (or defines) the difference in levels. It's not immediately obvious how such a system could be adapted for programming skills. But the idea of having levels, and having ways to measure one's level, is very interesting, and might be quite useful. Japanese (and probably other languages) have standardized tests[1] to determine one's language abilities. [1]: http://en.wikipedia.org/wiki/Japanese-Language_Proficiency_Test http://en.wikipedia.org/wiki/Japanese-Language_Proficiency_T... You don't take a test to determine your level. Rather, you take a test designed for a particular level. You either pass or fail. N5 is the lowest level (beginner) N1 is the strongest level (near-native or even better maybe) When you feel you are at say, level N3, you take the N3 test, and if you pass, then you can say that you have passed the N3 test. If you fail, you just fail. It doesn't mean you are N4 or N5. (As far as I know - I might actually be wrong here). I wonder if some standardized tests can be, in principle, constructed to measure programming abilities. The tests must be crafted such that, even if all the questions are known to the public, then studying for the test is the same thing as studying to build up your skill level.

pjmorris 12y ago
Should you measure the worker, or measure the work? Anyone who can build reliable systems using unreliable parts (like humans) will have an edge. EDIT: If you're going to measure the worker's programming concept knowledge, Alison Tew's work seems to be at the leading edge. [1] 'Developing a Validated Assessment of Fundamental CS1 Concepts', Tew, Guzdial, http://dl.acm.org/citation.cfm?id=1734297 http://dl.acm.org/citation.cfm?id=1734297
- hasenj 12y ago
  I'm more interested in measuring abilities relevant directly to software as a profession (or craftsmanship), such as: how well can you architecture a system to solve a complex problem?
  - eru 12y ago
    If you can shine a new light on the problem to make it simple instead, than you can get away with a simpler architecture.
  - wellpast 12y ago
    This is important: you can separate handling complexity from the profession of building architecturally sound systems. We can teach people how to do math correctly. That doesn't guarantee that they will be able to solve arbitrarily complex problems. But to solve math problems, you MUST at least have the skill sets. So there is a test to measure do you have the skill sets. How much one is willing and/or able to handle complexity is a separate thing to consider.
- lsiebert 12y ago
  Thanks, that was a very interesting paper (though I had to google to find a copy).
saidajigumi 12y ago
> I wonder if some standardized tests can be, in principle, constructed to measure programming abilities. This absolutely misses the point of the article. It is definitively not about programming abilities, or speed of these abilities. Many authors have hit TFA's points from different angles. For example, Tom Demarco's excellent Slack[1] introduces uses of "efficiency" (what speed is the organization moving at?) versus "effectiveness" (is the organization moving in the right direction?). DeMarco uses these to illustrate that many organizations optimize for the wrong parameters: they want to move at breakneck speed ("efficiency", "butts in seats", etc.), but in doing so trade off their vital strategic ability to think, design, to steer the ship! In an organization without slack (briefly, the ability of knowledge workers in an organization to engage in vital reflective tasks, a key part of TFA's design process), everyone's so focused on Getting There that no one remains to decide if There is the right place to be going. This can and does result in everything from classic technical debt to strategic business failure. DeMarco describes a pile of organizational anti-patterns that stem from this philosophical damage. I like that TFA's author connects these concepts explicitly to more recent thinking on the importance of design process. [1] http://www.amazon.com/Slack-Getting-Burnout-Busywork-Efficiency/dp/0932633617 http://www.amazon.com/Slack-Getting-Burnout-Busywork-Efficie...
lectrick 12y ago
It is the MOST trivial thing to measure performance in a Go game. It is the LEAST trivial thing to measure the "code quality"/"unit of time" metric in a programmer. For example, you might bang out an implementation that looks fine, but 1 guy will say "that will become unmaintainable in 1 year" or "that will be a problem if we ever switch databases" or whatever and sure enough, a year later, the team has to do that... and that 1 guy was fucking RIGHT. Who's to say how you measure such a thing? The longer you spend in a career that has ANY creative component, the more you will come to loathe the importance placed on "performance evaluations." In fact, I'd argue that the more advanced you are in those creative jobs, the SLOWER and BETTER you work... the problem is that the newbies won't be able to recognize the "better" portion.
- jodrellblank 12y ago
  "that will become unmaintainable in 1 year" or "that will be a problem if we ever switch databases" or whatever and sure enough, a year later, the team has to do that... and that 1 guy was fucking RIGHT. And sometimes YAGNI, we'll cross that bridge when we come to it, nice problem to have, good enough is good enough, do the simplest thing that could possibly work, premature optimization, perfect is the enemy of good, grass is always greener somewhere else, better a bird in the hand than two in the bush, he's right but it's still not worth the cost right now, etc.
  - hvidgaard 12y ago
    That is a perfect mindset to have to the problem of the type "that will be a problem if we ever switch databases". However, "that will become unmaintainable in 1 year" is a "bridge" you want to cross sooner rather than later. If you have a deadline to meet, fine, but come back and fix it asap if the software is something that needs to be maintained. I really cannot explain it well, it's just something most developers realize after a while when they've had to maintain an old codebase. They key point however is, that one guys insight shouldn't be ignored. You can take the words of wisdom and proceed to not do anything about it, but at least you know the cost down the road. All experienced managers I know personally, know the cost of umaintainable software and are more than willing to do something about it if ressources allow it. They need to, otherwise it's their ass that is in the line of fire when it cost a factor 10 to implement a new feature and bugs creep up at the customer time after time, even though they spend a factor 10 more on QA.

by hasenj 12y ago

pjmorris 12y ago

Should you measure the worker, or measure the work? Anyone who can build reliable systems using unreliable parts (like humans) will have an edge. EDIT: If you're going to measure the worker's programming concept knowledge, Alison Tew's work seems to be at the leading edge. [1] 'Developing a Validated Assessment of Fundamental CS1 Concepts', Tew, Guzdial, http://dl.acm.org/citation.cfm?id=1734297 http://dl.acm.org/citation.cfm?id=1734297

hasenj 12y ago
I'm more interested in measuring abilities relevant directly to software as a profession (or craftsmanship), such as: how well can you architecture a system to solve a complex problem?
- eru 12y ago
  If you can shine a new light on the problem to make it simple instead, than you can get away with a simpler architecture.
- wellpast 12y ago
  This is important: you can separate handling complexity from the profession of building architecturally sound systems. We can teach people how to do math correctly. That doesn't guarantee that they will be able to solve arbitrarily complex problems. But to solve math problems, you MUST at least have the skill sets. So there is a test to measure do you have the skill sets. How much one is willing and/or able to handle complexity is a separate thing to consider.
lsiebert 12y ago
Thanks, that was a very interesting paper (though I had to google to find a copy).

saidajigumi 12y ago

> I wonder if some standardized tests can be, in principle, constructed to measure programming abilities. This absolutely misses the point of the article. It is definitively not about programming abilities, or speed of these abilities. Many authors have hit TFA's points from different angles. For example, Tom Demarco's excellent Slack[1] introduces uses of "efficiency" (what speed is the organization moving at?) versus "effectiveness" (is the organization moving in the right direction?). DeMarco uses these to illustrate that many organizations optimize for the wrong parameters: they want to move at breakneck speed ("efficiency", "butts in seats", etc.), but in doing so trade off their vital strategic ability to think, design, to steer the ship! In an organization without slack (briefly, the ability of knowledge workers in an organization to engage in vital reflective tasks, a key part of TFA's design process), everyone's so focused on Getting There that no one remains to decide if There is the right place to be going. This can and does result in everything from classic technical debt to strategic business failure. DeMarco describes a pile of organizational anti-patterns that stem from this philosophical damage. I like that TFA's author connects these concepts explicitly to more recent thinking on the importance of design process. [1] http://www.amazon.com/Slack-Getting-Burnout-Busywork-Efficiency/dp/0932633617 http://www.amazon.com/Slack-Getting-Burnout-Busywork-Efficie...

lectrick 12y ago

It is the MOST trivial thing to measure performance in a Go game. It is the LEAST trivial thing to measure the "code quality"/"unit of time" metric in a programmer. For example, you might bang out an implementation that looks fine, but 1 guy will say "that will become unmaintainable in 1 year" or "that will be a problem if we ever switch databases" or whatever and sure enough, a year later, the team has to do that... and that 1 guy was fucking RIGHT. Who's to say how you measure such a thing? The longer you spend in a career that has ANY creative component, the more you will come to loathe the importance placed on "performance evaluations." In fact, I'd argue that the more advanced you are in those creative jobs, the SLOWER and BETTER you work... the problem is that the newbies won't be able to recognize the "better" portion.

jodrellblank 12y ago
"that will become unmaintainable in 1 year" or "that will be a problem if we ever switch databases" or whatever and sure enough, a year later, the team has to do that... and that 1 guy was fucking RIGHT. And sometimes YAGNI, we'll cross that bridge when we come to it, nice problem to have, good enough is good enough, do the simplest thing that could possibly work, premature optimization, perfect is the enemy of good, grass is always greener somewhere else, better a bird in the hand than two in the bush, he's right but it's still not worth the cost right now, etc.
- hvidgaard 12y ago
  That is a perfect mindset to have to the problem of the type "that will be a problem if we ever switch databases". However, "that will become unmaintainable in 1 year" is a "bridge" you want to cross sooner rather than later. If you have a deadline to meet, fine, but come back and fix it asap if the software is something that needs to be maintained. I really cannot explain it well, it's just something most developers realize after a while when they've had to maintain an old codebase. They key point however is, that one guys insight shouldn't be ignored. You can take the words of wisdom and proceed to not do anything about it, but at least you know the cost down the road. All experienced managers I know personally, know the cost of umaintainable software and are more than willing to do something about it if ressources allow it. They need to, otherwise it's their ass that is in the line of fire when it cost a factor 10 to implement a new feature and bugs creep up at the customer time after time, even though they spend a factor 10 more on QA.