9 ms·

Measuring software engineering competency

129 points by ferroman 9y ago

jt2190 9y ago
A little additional context: Joel's "test" was just a quick-and-dirty way for a job candidate to assess whether or not a company had competent software engineering practices. Assessing your own team's practices could (and should) go far more in depth, since all the messy details are right there. The spirit of this article is good, but I wonder if we can formulate a good test that applies universally to all teams and carries enough detail to help that team improve.
quantumhobbit 9y ago
I'd settle for a place that stuck to the original Joel Test. After all most of the CICD stuff in this list is covered by his original 1-step builds rule. If you have that then scripting cicd stuff is trivial. Similarly his test covers testing. Everything else in here is just disguised Scrum.
- ferroman 9y ago
  True, but not much team follow these, in practice. We just want to make it more specific and modern.
  - vog 9y ago
    > True, but not much team follow these Isn't this a very good indicator that the Joel test is still sufficient? Only once almost all companies pass the Joel test with 12/12 points [1], a modernization is needed to distinguish further between multiple offerings. [1] or 11/12 points, see https://news.ycombinator.com/item?id=14078069 https://news.ycombinator.com/item?id=14078069 EDIT: Fix typo.
    - ferroman 9y ago
      It is relevant, but we try to make it more specific.
      - vog 9y ago
        Sorry, typo. I meant "sufficient", not "relevant".
  - ska 9y ago
    Is that a reflection on the test, or the teams?
    - ferroman 9y ago
      Teams, of course. The goal was to measure it, and we tried to be more specific.
kensai 9y ago
If you really want to improve the Joel Test, fine. The suggestions are even right. But please, KISS. The original test had 12 items, this one 20. This should be the upper limit, more or less, for any improved version. Otherwise it is probably too long or detailed to be of practical use.
- ferroman 9y ago
  Thanks for feedback. We decide that more detailed test - more precise measure, but we still try to keep the list as short as possible.
- solatic 9y ago
  It's also way too specific and biased. Rules like "do you have end-to-end integration tests?" aren't always obligations for all teams, whether that's because your team is doing embedded work or because you're doing something better (like consumer-driven contracts). Other rules like "do you have a primary communication channel?" are even at times counter-productive (in particular, when dealing with a variety of customers who have their own preferred methods of communication, which you must accommodate). Daily status meetings are sometimes unnecessary, if you have a small enough team sitting in its own room or more tightly involve team members in planning and review practices, and indeed daily status meetings often clash with more important practices like flexible scheduling and not interrupting flow.
TeeWEE 9y ago
Do you have automated end-to-end tests in DSL? Seriously? What is considered a DSL?
- ryanbrunner 9y ago
  And what's more, why does it matter? I assume they mean something like Cucumber, which, while I'm OK with, doesn't really add any benefit as well organized tests written in the language your code is in.
  - ferroman 9y ago
    Because it allow to keep the domain logic somewhere. In these days people don't do documentation, or documentation is always outdated. Having high-level tests in DSL will do more than just tests - it give you information how your application behave from the user perspective, in more general sense. So you will have less issues when new people join to the project, or when project owner changes. And it force you to focus goal during feature implementation. From my experience, new features often have behavior that are not oblivious, and some times conflict with other application features logic. These tests allow to see these conflicts before implementation.
    - ryanbrunner 9y ago
      I disagree, but I get where you're coming from - which is exactly what the problem with this list is. The Joel Test was great because everything on it was universally accepted as something that every developer would want to happen where they worked. With this, different developers are going to have different opinions of many things on the list, making the "score" lose it's usefulness, since now every time a company has less than a perfect score, I need to figure out why (is it because they don't have a library? I don't care so much about that. Is it because they don't have CI? I care a lot about that.)
  - benmmurphy 9y ago
    i'm not a fan of cucumber and you can achieve the same results by just reusing code. a big problem i have is it ends up introducing pointless indirection. defining steps in different files from the feature being tested even if they are only used in that single feature file is unnecessary abstraction. having to grep for step syntax in a project to try and find the code that is implementing a step is insane.
Androider 9y ago
A lot of companies get the basic source control, builds, bug tracking and writing code during interviews parts right, but tend to skimp on these aspects of the Joel test: - Do you fix bugs before writing new code? - Do programmers have quiet working conditions? - Do you use the best tools money can buy?
- vog 9y ago
  While I agree with most questions, I always found number 9 to be a bit strange: > 9. Do you use the best tools money can buy? I hope I don't do too much injustice on Joel, as I always loved to read his writings back when his "Joel on Software" blog was active. However, this item on the list always sounded to me as an attempt to promote their FogBugz tool, not as an objective advice. Recognizing there are many excellent Free Software tools, specially in the software development area, I'd rephrase it is as: > 9a. Do you use the best tools available? or maybe: > 9b. Do you invest in your tools? which means, depending on the exact tool, one or more of: - buying a proprietary tool - using a Free Software tool, and donating money to the project - using a Free Software tool, and providing bug fixes and/or new features - having one or more team members dedicated to improve the tooling and infrastructure
  - GavinMcG 9y ago
    I always took "the best money can buy" to be an idiom for "the best available". I don't think he was advocating for paid tools literally whenever possible.
  - Androider 9y ago
    Frequently the objectively best tool is Free or open source software (which doesn't mean it's priced at $0, although often it will be). But many times it's not, and that's when companies can become extremely penny wise and pound foolish. Hiring someone to exclusively babysit a Jenkins instance is incredibly expensive. Paying for Travis CI/Codeship/Gitlab CI is really cheap in comparison. Having developers fill out purchasing orders and waiting for software or hardware is very expensive. I like to call it the "IntelliJ test", can I requisition IntelliJ ($499) and have it the same day (week? month?) or is the company going to flinch, hem and haw at the absolutely inconsequential price of the software in comparison to the expensive developer time they're paying for.
ryanbrunner 9y ago
This has 3 big problems that make it a poor substitute for the Joel Test: there's way too many questions, some of the questions don't have a universally accepted "good" answer, and questions have too much ambiguity and wiggle-room. One nice factor of the Joel Test (not that I saw it being used in reality - but as a mental model anyways), was that you could easily categorize companies into places you want to work or places you don't want to work. A perfect score? You want to work there. More than 2 things they don't do? You don't want to work there. 1 thing? Maybe look into it and see how important it is to you. With this, your scores could be all over the map. What's more, the questions a company misses on might be ones that aren't that important to you (having a library), or even where a 'no' might be preferable to you (daily stand-up). Even once you get past all that, many questions aren't easily answered objectively. What's a short iteration? I've worked in places that touted 2 weeks as a remarkably short iteration, and others who bemoaned how long that was.
- jcadam 9y ago
  > A perfect score? You want to work there. More than 2 things they don't do? You don't want to work there. 1 thing? Maybe look into it and see how important it is to you. That's all well and good if you have the luxury of picking and choosing from multiple offers. Here in the real world (i.e., not in SV), getting a decent offer (if you're not entry level) that's at least equal to your current pay generally takes 6 months - 1 year of hard interviewing. If I demanded a prospective employer scored even 50% on the Joel Test, I'd be perpetually unemployed. Which is probably why employers generally get away with providing sucktastic working environments for software developers. I find it especially disheartening that the only one of Joel's 12 'tests' that's pretty much a universal 'yes' these days is uses source control.
  - ryanbrunner 9y ago
    I'm not from SV, but I am from a tech hub, so I'm sure our experiences are pretty different. Regardless of that, adding granularity and ambiguity to the score doesn't help much in your case either (particularly since things like source control were removed, probably because "it's a given" where that might not be the case outside of startups)
NumberSix 9y ago
Conspicuous by its absence is: Does your software work? These criteria are all rituals and processes, rather than the end result.
- ryanbrunner 9y ago
  They're signals that indicate a well-functioning development team, that can be researched with little effort and answered objectively. "Does your software work?" is almost impossible to answer objectively, and doesn't help you determine if the software is going to work 2 years from now (which a good deal of development best practices work to achieve). You might as well replace the test with "Is this company awesome?"
  - NumberSix 9y ago
    In general, the way to objectively determine if software works is to give it to those pesky end users, have them use it, and tell you if it works or not, and how well it works. Lacking end users, have testers and QA folks who play the role of end users evaluate it. In some areas of software development, such as heavy duty algorithmic/mathematical programs such as encryption, video compression, computer graphics, there are pretty rigorous objective ways to measure whether the program works and how well, without human testers or QA people. Usually best to do both even in these cases, just in case there are some subtle issues that the performance metrics don't capture. On the other hand, predicting the future is noted for being rather hard. Predicting correctly whether a program will need to be changed in two years, in what way, and whether the program can in fact be changed easily all two years in the future is speculation, a matter of opinion, rarely objective at all.
    - ryanbrunner 9y ago
      You're right that there are classes of application that can be said to "work" objectively. For the majority of business facing SaaS applications (to name an example), "working" is an elusive target. If we gave it to those pesky end users, and there's more than 5 of them, I guarantee you'd hear multiple answers to how well it works certainly, and even if it works. I'm not saying that looking at how well your product works for people isn't a noble endeavour or anything. For the purpose of what this is supposed to be - an easily obtainable, objective measure of what it's like to work for a company, it's horrible.
mlashcorp 9y ago
I may be in the minority, but I find the 50% unit test coverage not useful, and sometimes harmful. Caveat emptor - depends a lot on the project, and how often you are changing the code and/or the complexity of said code.
- ferroman 9y ago
  What do you mean? Is it's too low or too high?
  - mlashcorp 9y ago
    In my experience it's too high. I see a lot of unit test code that doesn't do anything except add complexity. But again, I guess this will depend on the nature of the project
  - quantumhobbit 9y ago
    It is less a question of whether the percentage is correct than whether the tests are useful. I've seen plenty of useless tests (testing getters and setters in Java) that assert nothing related to the codes functionality but exist solely to boost coverage. Which is why asserting a strict coverage percentage is dangerous. Better to just do real TDD in the first place.
    - jghn 9y ago
      This. Slavish fetishization of a specific code coverage target is indicative of an underlying problem, IMO and that problem is far greater than one having relatively low code coverage. It is far better IMO to go in with an understanding of where your potential hot spots are than simply adding a test to everything. Sure, in an ideal world we'd have 100% coverage of everything but this field is about tradeoffs and sometimes writing tests simply isn't worth the time it takes to have written them in the long run.
- ck425 9y ago
  Same here. Coverage is no indication of quality, and lack of coverage isn't necessarily an indicator of lacking quality.
djb_hackernews 9y ago
The problem with this test and the Joel test is an employer can check all of the boxes (essentially do you follow modern software practices that were revolutionary 20 years ago, and are you "Agile") and it can still result in a toxic or less than optimal environment. I did like the questions around OSS and sharing expertise. I'd like to see more questions that address recruitment anti patterns (diversity, agism, disclosing previous salary, etc) and tech organization anti patterns (an actual career path on par with management, non transparent equity grants, etc) Like, what would the questions be if even, say, Google didn't look so good if it answered them.
martijn_himself 9y ago
It seems to me that more and more ceremony is being added to software development which distracts from the actual work. This only benefits 2 groups of people: people that don't like the actual work but still want to fulfill a role in the process, and the agile 'industry'.
- humanrebar 9y ago
  > ...distracts from the actual work... YMMV here. I've much more often encountered situations where I had to interview the maintainer of a repo (if there even was an official one) to find out how to contribute, whether they were even interested in contributions, and how to make sure my change didn't break anything. A lot of these tools, processes, and special words are as much about good passive communication as anything else. That being said, tooling that doesn't fit development use cases is often worse than no tooling (presuming devs can make their own productivity scripts as needed).
- deleted 9y ago
  [deleted]
- s73ver 9y ago
  Some of that is needed, though. If you're going to just shut one or two people in a room for 9 months, then you probably don't need it. But today usually you're going to have bigger teams, and the business is going to need more visibility into the project, while also giving developers a degree of autonomy and ownership of it. When the business is willing to let it work, Agile can be kinda nice. But if the business doesn't buy in, if they keep interrupting, changing priorities and tasks mid-sprint, then it's just going to make everyone miserable.
  - marcosdumay 9y ago
    > Some of that is needed, though. The most important thing you need to make sure is that developers talk with end users. That is one of the biggest point of Agile. Now, if you look again at the article, you'll notice this one interaction is completely missing. Not only that, but Scrum de-emphasize it too by creating middle-men, and most formal "Agile" methodologies don't even think about it.
throwaway729 9y ago
A Joel List is a quick-and-dirty list you can use to assess the competence of an organization. I don't think this list achieves that goal. #1 and #2 (CICD) are fair additions to Joel's list, but I'd argue are already encapsulated by "do you make daily builds?". In most shops, if you make daily builds, then you CICD. #3 = Joel's #4 #4,#5,#8,#10,#11,#12,#14,#15 are all "Do you SCRUM/TDD?". If that's the kind of place you're looking for, great. But there are many competent code-oriented organizations that do not SCRUM. So these don't really belong on a Joel List. (Also, "We don’t know the better way to make sure that code does what it’s supposed to, then to have another code [author means unit tests] that runs it and check results" just isn't true. We know better ways, and sometimes they're even relevant to a list like this. "Do you use any form of static or dynamic analysis (e.g., types, valgrind, quick-check style tools, linters, etc.)" is on my personal "Joel Test".) That leaves "do you have a library?". IMO work-place libraries are close to useless as signals (everyone has one), and rarely useful in practice (unless you're curious how PHP code was written in 2003 or really want to brush up on complexity theory). As an aside, it's kind of depressing to me that we still make these lists. Back in the 90's, software engineering was still a relatively young craft with relatively few experts. Joel was part of a surprisingly small group of people who: 1) had a career's worth of experience developing software for micro-computers in high level languages; and 2) had deep and successful experiences across several organization roles in different types of organizations (coder, manager at MSFT, CEO at Fog Creek). The existence of managers who were in charge of software engineers but had no engineering experience wasn't surprising at all, given the youth of the field. Hence the Joel Test. The world is a very different place today. There are a lot of people with this level of experience. Joel Tests aren't ubiquitous in other engineering domains, and hopefully they'll eventually die out in software as well. Not because the items on them aren't important, but because experienced Engineers manage Engineers.
- ferroman 9y ago
  You can generalize it, but that was a point to make the list more specific. And yes, world is a very different place today. The number of SW engineers doubles every year, so, at least half for them are new. We are engineers and we should try to measure or competency, аnd should try to systematize the things that we use. Of course, this list is not something absolutely universal, but we should at least try think about standards we that we want to meet.
obstinate 9y ago
Never even heard of half the terms in "obligations". Somebody should tell my employer that I'm not competent.
- samtolife 9y ago
  Eventually some one will :)
  - mcguire 9y ago
    Probably the new intern with the 'information technology' major.
JCDenton2052 9y ago
Such tests are not very informative when they are disconnected from the type of companies/industries one is looking for. This is a criticism I have of the Joel test, too, since at the time he wrote it it seemed to me a good way to evaluate prospective software houses. After all, he worked in one. Yet I dare say most of us will probably spend at least part of our careers working in in-house IT departments. Different rules. It is not unlikely to find companies that fulfill all requirements, although they will likely know how attractive their working environments are and will filter candidates accordingly. An interview I had with such a small-sized software house two months ago confirmed this. I gave them the Joel test, which they had never heard of before, and they scored perfect. Dedicated testers, usability testing, quiet working environments (like a library, the team lead said; no need for headphones). Predictably, they were extremely picky as to who they let in. The ones much less likely to get good test scores? 1) Government IT, by and large 2) IT for any non-tech company less than a certain size. 3) Non-tech corporations (and even some tech ones). One notable one I was aware of used excel for bug tracking, was full of red tape and their main technical test was a 20 question multiple choice.
- s73ver 9y ago
  I'm not quite so sure what your point is. The examples you gave of places that don't do so well on the Joel test still sound like places most of us would not want to work.
BeetleB 9y ago
>Do you contribute to Open Source? Of all the Merits, this is one I disagree with. It's merit is primarily "joining like minded folks" (i.e. cult) than any inherent merit in and of itself. How many great developers do you know who do not have this "merit"?
- azrazalea 9y ago
  I know way more great developers who contribute to open source than great developers that don't.
  - Dayshine 9y ago
    Of course you do, they publicize themselves by contributing to open source...
- djb_hackernews 9y ago
  I think you are misreading this question. It is a merit of the organization and I think gets at the issue of lots of organizations using OSS, but very few of them actually contributing back. What this requires of the organization is to acknowledge that they need to contribute if they want to benefit long term and thus they need to allow and encourage their engineers to participate. If employers were encouraging or at least allowed it you'd see a lot more great developers have this merit, but for most medium to large organizations it's a one way street with OSS and prohibit their engineers from open sourcing projects or contributing to projects the organization depends on.
mcguire 9y ago
A) Actually, we have plenty of ways of measuring software engineering competence.[1][2][3] B) Weirdly, all of them are heavily process focused and none of them have much interest in the ability to write functioning code, because C) Software engineering, as a field, is built on the belief that anyone can write software if properly managed.[4] [1] http://cmmiinstitute.com/ http://cmmiinstitute.com/ [2] http://www.sei.cmu.edu/certification/ http://www.sei.cmu.edu/certification/ [3] https://www.computer.org/web/education/certifications https://www.computer.org/web/education/certifications [4] and that would make software development much cheaper.
- mannykannot 9y ago
  Re C): Indeed - and the belief extends to the proposition that to properly manage it, you do not need much understanding of what the detailed design and coding aspects of development actually entail, so long as you know Software Engineering.
  - mcguire 9y ago
    That is an extension of the old 'managers need to know how to manage, not how to do things.'
- BatFastard 9y ago
  Indeed anyone can write software. However writing good, maintainable, scalable software is a totally different thing. There are so many skills (someone posted a skill matrix recently which I liked). I taught martial arts for many years, and I can honestly say that I can teach martial arts to anyone. However 98% of those learning will suck at it. They don't have the aptitude, the dedication, or pain tolerance.
mbesto 9y ago
I do technology due diligence on behalf of investors for a living, so I live and breathe this type of stuff on a weekly basis. All companies build software differently. Some have automatic deployment, some don't. Some have strong testing procedures, some don't. Just because a company doesn't use a CI, doesn't necessarily make them "worse". It's just an indifference to the indoctrination of the "SV mindset". The more important answers are not binary yes/no by rather "why aren't using a CI". Common answers are: - I'm not sure what CI is - We don't have enough unit tests to justify it - We're a small team and it doesn't really justify the effort to setup - We're working on setting up and should be live in the next 3 months You can tell a lot about engineering competency and leadership from those answers.
- nudpiedo 9y ago
  Not sure why you got downvoted, you are clearly speaking out of your experience working with different teams and expoisng real answers which many people could say loud every second day.
- deegles 9y ago
  Any advice on getting into this field? Seems like it would be pretty rewarding (mentally) work.
  - mbesto 9y ago
    Honestly - I stumbled into it. I really don't have any good advice other than know the right people.
- wolfgke 9y ago
  My personal answer rather is: One is working on a kind of software with additional safety or security requirements, where CI would be a really bad idea. This does not contradict to the idea that methods that are rather necessary for CI, such as really high test coverage (this is typically even a requirement for such a kind of software), automatic building (can improve productivity a lot) often also make sense in such an environment.
kazinator 9y ago
Do you follow the shibboleths of my religious sect?
- deleted 9y ago
  [deleted]
deleted 9y ago
[deleted]
dwheeler 9y ago
If you're looking for a set of basic criteria for well-run open source software projects, please check out the CII Best Practices Badge: https://bestpractices.coreinfrastructure.org/ https://bestpractices.coreinfrastructure.org/ Full disclosure: I lead the project. Constructive comments very welcome!
jondubois 9y ago
This test completely misses what it means to be a senior (effective) engineer. The real difference between a mid-level engineer and a senior engineer is that the mid-level engineer will mechanically apply the same strategies to all projects without thinking - All the points mentioned in this article; CI, one-step deployment, daily status check-in meetings, etc, etc... are in fact not necessary for ALL projects. The quote "Those who only have a hammer tend to see every problem as a nail" is a good summary of the junior/mid-level mindset. I don't know the author, but based on the rigidity of the article, I would guess that they've only worked for big companies. I would argue that a most of these rules are only effective in the context of a very large company; in literally every other context, many of these rules are inefficient. Big companies are all about risk mitigation; they are willing to sacrifice speed and agility in exchange for stability, certainty and visibility but this is actually a luxury that only big companies can afford and should not be taken as a rule of thumb.
Profragile 9y ago
-7 thanks. I would do absolutely anything to avoid working with whoever scores very high in this test.
Silhouette 9y ago
I think I get -9 for one current project, though I'm not sure because I don't understand several of the weird ones. Then again, the client likes that project, because their customers also like it. It solves a problem for them that no-one had solved in a similar way before. I don't think we've had a single major bug reported against that part of the system by any customer in over five years, and typically that includes a multi-month lab evaluation by each customer before deployment. So, does the development process on that project suck or not? :-)
suhith 9y ago
It would be great to have some compiled data on various companies/teams and the obligations + merits that they do or do not meet.

129 points by ferroman 9y ago

A little additional context: Joel's "test" was just a quick-and-dirty way for a job candidate to assess whether or not a company had competent software engineering practices. Assessing your own team's practices could (and should) go far more in depth, since all the messy details are right there. The spirit of this article is good, but I wonder if we can formulate a good test that applies universally to all teams and carries enough detail to help that team improve.

quantumhobbit 9y ago

I'd settle for a place that stuck to the original Joel Test. After all most of the CICD stuff in this list is covered by his original 1-step builds rule. If you have that then scripting cicd stuff is trivial. Similarly his test covers testing. Everything else in here is just disguised Scrum.

ferroman 9y ago
True, but not much team follow these, in practice. We just want to make it more specific and modern.
- vog 9y ago
  > True, but not much team follow these Isn't this a very good indicator that the Joel test is still sufficient? Only once almost all companies pass the Joel test with 12/12 points [1], a modernization is needed to distinguish further between multiple offerings. [1] or 11/12 points, see https://news.ycombinator.com/item?id=14078069 https://news.ycombinator.com/item?id=14078069 EDIT: Fix typo.
  - ferroman 9y ago
    It is relevant, but we try to make it more specific.
    - vog 9y ago
      Sorry, typo. I meant "sufficient", not "relevant".
- ska 9y ago
  Is that a reflection on the test, or the teams?
  - ferroman 9y ago
    Teams, of course. The goal was to measure it, and we tried to be more specific.

kensai 9y ago

If you really want to improve the Joel Test, fine. The suggestions are even right. But please, KISS. The original test had 12 items, this one 20. This should be the upper limit, more or less, for any improved version. Otherwise it is probably too long or detailed to be of practical use.

ferroman 9y ago
Thanks for feedback. We decide that more detailed test - more precise measure, but we still try to keep the list as short as possible.
solatic 9y ago
It's also way too specific and biased. Rules like "do you have end-to-end integration tests?" aren't always obligations for all teams, whether that's because your team is doing embedded work or because you're doing something better (like consumer-driven contracts). Other rules like "do you have a primary communication channel?" are even at times counter-productive (in particular, when dealing with a variety of customers who have their own preferred methods of communication, which you must accommodate). Daily status meetings are sometimes unnecessary, if you have a small enough team sitting in its own room or more tightly involve team members in planning and review practices, and indeed daily status meetings often clash with more important practices like flexible scheduling and not interrupting flow.

TeeWEE 9y ago

Do you have automated end-to-end tests in DSL? Seriously? What is considered a DSL?

ryanbrunner 9y ago
And what's more, why does it matter? I assume they mean something like Cucumber, which, while I'm OK with, doesn't really add any benefit as well organized tests written in the language your code is in.
- ferroman 9y ago
  Because it allow to keep the domain logic somewhere. In these days people don't do documentation, or documentation is always outdated. Having high-level tests in DSL will do more than just tests - it give you information how your application behave from the user perspective, in more general sense. So you will have less issues when new people join to the project, or when project owner changes. And it force you to focus goal during feature implementation. From my experience, new features often have behavior that are not oblivious, and some times conflict with other application features logic. These tests allow to see these conflicts before implementation.
  - ryanbrunner 9y ago
    I disagree, but I get where you're coming from - which is exactly what the problem with this list is. The Joel Test was great because everything on it was universally accepted as something that every developer would want to happen where they worked. With this, different developers are going to have different opinions of many things on the list, making the "score" lose it's usefulness, since now every time a company has less than a perfect score, I need to figure out why (is it because they don't have a library? I don't care so much about that. Is it because they don't have CI? I care a lot about that.)
- benmmurphy 9y ago
  i'm not a fan of cucumber and you can achieve the same results by just reusing code. a big problem i have is it ends up introducing pointless indirection. defining steps in different files from the feature being tested even if they are only used in that single feature file is unnecessary abstraction. having to grep for step syntax in a project to try and find the code that is implementing a step is insane.

Androider 9y ago

A lot of companies get the basic source control, builds, bug tracking and writing code during interviews parts right, but tend to skimp on these aspects of the Joel test: - Do you fix bugs before writing new code? - Do programmers have quiet working conditions? - Do you use the best tools money can buy?

vog 9y ago
While I agree with most questions, I always found number 9 to be a bit strange: > 9. Do you use the best tools money can buy? I hope I don't do too much injustice on Joel, as I always loved to read his writings back when his "Joel on Software" blog was active. However, this item on the list always sounded to me as an attempt to promote their FogBugz tool, not as an objective advice. Recognizing there are many excellent Free Software tools, specially in the software development area, I'd rephrase it is as: > 9a. Do you use the best tools available? or maybe: > 9b. Do you invest in your tools? which means, depending on the exact tool, one or more of: - buying a proprietary tool - using a Free Software tool, and donating money to the project - using a Free Software tool, and providing bug fixes and/or new features - having one or more team members dedicated to improve the tooling and infrastructure
- GavinMcG 9y ago
  I always took "the best money can buy" to be an idiom for "the best available". I don't think he was advocating for paid tools literally whenever possible.
- Androider 9y ago
  Frequently the objectively best tool is Free or open source software (which doesn't mean it's priced at $0, although often it will be). But many times it's not, and that's when companies can become extremely penny wise and pound foolish. Hiring someone to exclusively babysit a Jenkins instance is incredibly expensive. Paying for Travis CI/Codeship/Gitlab CI is really cheap in comparison. Having developers fill out purchasing orders and waiting for software or hardware is very expensive. I like to call it the "IntelliJ test", can I requisition IntelliJ ($499) and have it the same day (week? month?) or is the company going to flinch, hem and haw at the absolutely inconsequential price of the software in comparison to the expensive developer time they're paying for.

ryanbrunner 9y ago

This has 3 big problems that make it a poor substitute for the Joel Test: there's way too many questions, some of the questions don't have a universally accepted "good" answer, and questions have too much ambiguity and wiggle-room. One nice factor of the Joel Test (not that I saw it being used in reality - but as a mental model anyways), was that you could easily categorize companies into places you want to work or places you don't want to work. A perfect score? You want to work there. More than 2 things they don't do? You don't want to work there. 1 thing? Maybe look into it and see how important it is to you. With this, your scores could be all over the map. What's more, the questions a company misses on might be ones that aren't that important to you (having a library), or even where a 'no' might be preferable to you (daily stand-up). Even once you get past all that, many questions aren't easily answered objectively. What's a short iteration? I've worked in places that touted 2 weeks as a remarkably short iteration, and others who bemoaned how long that was.

jcadam 9y ago
> A perfect score? You want to work there. More than 2 things they don't do? You don't want to work there. 1 thing? Maybe look into it and see how important it is to you. That's all well and good if you have the luxury of picking and choosing from multiple offers. Here in the real world (i.e., not in SV), getting a decent offer (if you're not entry level) that's at least equal to your current pay generally takes 6 months - 1 year of hard interviewing. If I demanded a prospective employer scored even 50% on the Joel Test, I'd be perpetually unemployed. Which is probably why employers generally get away with providing sucktastic working environments for software developers. I find it especially disheartening that the only one of Joel's 12 'tests' that's pretty much a universal 'yes' these days is uses source control.
- ryanbrunner 9y ago
  I'm not from SV, but I am from a tech hub, so I'm sure our experiences are pretty different. Regardless of that, adding granularity and ambiguity to the score doesn't help much in your case either (particularly since things like source control were removed, probably because "it's a given" where that might not be the case outside of startups)

NumberSix 9y ago

Conspicuous by its absence is: Does your software work? These criteria are all rituals and processes, rather than the end result.

ryanbrunner 9y ago
They're signals that indicate a well-functioning development team, that can be researched with little effort and answered objectively. "Does your software work?" is almost impossible to answer objectively, and doesn't help you determine if the software is going to work 2 years from now (which a good deal of development best practices work to achieve). You might as well replace the test with "Is this company awesome?"
- NumberSix 9y ago
  In general, the way to objectively determine if software works is to give it to those pesky end users, have them use it, and tell you if it works or not, and how well it works. Lacking end users, have testers and QA folks who play the role of end users evaluate it. In some areas of software development, such as heavy duty algorithmic/mathematical programs such as encryption, video compression, computer graphics, there are pretty rigorous objective ways to measure whether the program works and how well, without human testers or QA people. Usually best to do both even in these cases, just in case there are some subtle issues that the performance metrics don't capture. On the other hand, predicting the future is noted for being rather hard. Predicting correctly whether a program will need to be changed in two years, in what way, and whether the program can in fact be changed easily all two years in the future is speculation, a matter of opinion, rarely objective at all.
  - ryanbrunner 9y ago
    You're right that there are classes of application that can be said to "work" objectively. For the majority of business facing SaaS applications (to name an example), "working" is an elusive target. If we gave it to those pesky end users, and there's more than 5 of them, I guarantee you'd hear multiple answers to how well it works certainly, and even if it works. I'm not saying that looking at how well your product works for people isn't a noble endeavour or anything. For the purpose of what this is supposed to be - an easily obtainable, objective measure of what it's like to work for a company, it's horrible.

mlashcorp 9y ago

I may be in the minority, but I find the 50% unit test coverage not useful, and sometimes harmful. Caveat emptor - depends a lot on the project, and how often you are changing the code and/or the complexity of said code.

ferroman 9y ago
What do you mean? Is it's too low or too high?
- mlashcorp 9y ago
  In my experience it's too high. I see a lot of unit test code that doesn't do anything except add complexity. But again, I guess this will depend on the nature of the project
- quantumhobbit 9y ago
  It is less a question of whether the percentage is correct than whether the tests are useful. I've seen plenty of useless tests (testing getters and setters in Java) that assert nothing related to the codes functionality but exist solely to boost coverage. Which is why asserting a strict coverage percentage is dangerous. Better to just do real TDD in the first place.
  - jghn 9y ago
    This. Slavish fetishization of a specific code coverage target is indicative of an underlying problem, IMO and that problem is far greater than one having relatively low code coverage. It is far better IMO to go in with an understanding of where your potential hot spots are than simply adding a test to everything. Sure, in an ideal world we'd have 100% coverage of everything but this field is about tradeoffs and sometimes writing tests simply isn't worth the time it takes to have written them in the long run.
ck425 9y ago
Same here. Coverage is no indication of quality, and lack of coverage isn't necessarily an indicator of lacking quality.

djb_hackernews 9y ago

The problem with this test and the Joel test is an employer can check all of the boxes (essentially do you follow modern software practices that were revolutionary 20 years ago, and are you "Agile") and it can still result in a toxic or less than optimal environment. I did like the questions around OSS and sharing expertise. I'd like to see more questions that address recruitment anti patterns (diversity, agism, disclosing previous salary, etc) and tech organization anti patterns (an actual career path on par with management, non transparent equity grants, etc) Like, what would the questions be if even, say, Google didn't look so good if it answered them.

martijn_himself 9y ago

It seems to me that more and more ceremony is being added to software development which distracts from the actual work. This only benefits 2 groups of people: people that don't like the actual work but still want to fulfill a role in the process, and the agile 'industry'.

humanrebar 9y ago
> ...distracts from the actual work... YMMV here. I've much more often encountered situations where I had to interview the maintainer of a repo (if there even was an official one) to find out how to contribute, whether they were even interested in contributions, and how to make sure my change didn't break anything. A lot of these tools, processes, and special words are as much about good passive communication as anything else. That being said, tooling that doesn't fit development use cases is often worse than no tooling (presuming devs can make their own productivity scripts as needed).
deleted 9y ago
[deleted]
s73ver 9y ago
Some of that is needed, though. If you're going to just shut one or two people in a room for 9 months, then you probably don't need it. But today usually you're going to have bigger teams, and the business is going to need more visibility into the project, while also giving developers a degree of autonomy and ownership of it. When the business is willing to let it work, Agile can be kinda nice. But if the business doesn't buy in, if they keep interrupting, changing priorities and tasks mid-sprint, then it's just going to make everyone miserable.
- marcosdumay 9y ago
  > Some of that is needed, though. The most important thing you need to make sure is that developers talk with end users. That is one of the biggest point of Agile. Now, if you look again at the article, you'll notice this one interaction is completely missing. Not only that, but Scrum de-emphasize it too by creating middle-men, and most formal "Agile" methodologies don't even think about it.

throwaway729 9y ago

A Joel List is a quick-and-dirty list you can use to assess the competence of an organization. I don't think this list achieves that goal. #1 and #2 (CICD) are fair additions to Joel's list, but I'd argue are already encapsulated by "do you make daily builds?". In most shops, if you make daily builds, then you CICD. #3 = Joel's #4 #4,#5,#8,#10,#11,#12,#14,#15 are all "Do you SCRUM/TDD?". If that's the kind of place you're looking for, great. But there are many competent code-oriented organizations that do not SCRUM. So these don't really belong on a Joel List. (Also, "We don’t know the better way to make sure that code does what it’s supposed to, then to have another code [author means unit tests] that runs it and check results" just isn't true. We know better ways, and sometimes they're even relevant to a list like this. "Do you use any form of static or dynamic analysis (e.g., types, valgrind, quick-check style tools, linters, etc.)" is on my personal "Joel Test".) That leaves "do you have a library?". IMO work-place libraries are close to useless as signals (everyone has one), and rarely useful in practice (unless you're curious how PHP code was written in 2003 or really want to brush up on complexity theory). As an aside, it's kind of depressing to me that we still make these lists. Back in the 90's, software engineering was still a relatively young craft with relatively few experts. Joel was part of a surprisingly small group of people who: 1) had a career's worth of experience developing software for micro-computers in high level languages; and 2) had deep and successful experiences across several organization roles in different types of organizations (coder, manager at MSFT, CEO at Fog Creek). The existence of managers who were in charge of software engineers but had no engineering experience wasn't surprising at all, given the youth of the field. Hence the Joel Test. The world is a very different place today. There are a lot of people with this level of experience. Joel Tests aren't ubiquitous in other engineering domains, and hopefully they'll eventually die out in software as well. Not because the items on them aren't important, but because experienced Engineers manage Engineers.

ferroman 9y ago
You can generalize it, but that was a point to make the list more specific. And yes, world is a very different place today. The number of SW engineers doubles every year, so, at least half for them are new. We are engineers and we should try to measure or competency, аnd should try to systematize the things that we use. Of course, this list is not something absolutely universal, but we should at least try think about standards we that we want to meet.

obstinate 9y ago

Never even heard of half the terms in "obligations". Somebody should tell my employer that I'm not competent.

samtolife 9y ago
Eventually some one will :)
- mcguire 9y ago
  Probably the new intern with the 'information technology' major.

JCDenton2052 9y ago

Such tests are not very informative when they are disconnected from the type of companies/industries one is looking for. This is a criticism I have of the Joel test, too, since at the time he wrote it it seemed to me a good way to evaluate prospective software houses. After all, he worked in one. Yet I dare say most of us will probably spend at least part of our careers working in in-house IT departments. Different rules. It is not unlikely to find companies that fulfill all requirements, although they will likely know how attractive their working environments are and will filter candidates accordingly. An interview I had with such a small-sized software house two months ago confirmed this. I gave them the Joel test, which they had never heard of before, and they scored perfect. Dedicated testers, usability testing, quiet working environments (like a library, the team lead said; no need for headphones). Predictably, they were extremely picky as to who they let in. The ones much less likely to get good test scores? 1) Government IT, by and large 2) IT for any non-tech company less than a certain size. 3) Non-tech corporations (and even some tech ones). One notable one I was aware of used excel for bug tracking, was full of red tape and their main technical test was a 20 question multiple choice.

s73ver 9y ago
I'm not quite so sure what your point is. The examples you gave of places that don't do so well on the Joel test still sound like places most of us would not want to work.

BeetleB 9y ago

>Do you contribute to Open Source? Of all the Merits, this is one I disagree with. It's merit is primarily "joining like minded folks" (i.e. cult) than any inherent merit in and of itself. How many great developers do you know who do not have this "merit"?

azrazalea 9y ago
I know way more great developers who contribute to open source than great developers that don't.
- Dayshine 9y ago
  Of course you do, they publicize themselves by contributing to open source...
djb_hackernews 9y ago
I think you are misreading this question. It is a merit of the organization and I think gets at the issue of lots of organizations using OSS, but very few of them actually contributing back. What this requires of the organization is to acknowledge that they need to contribute if they want to benefit long term and thus they need to allow and encourage their engineers to participate. If employers were encouraging or at least allowed it you'd see a lot more great developers have this merit, but for most medium to large organizations it's a one way street with OSS and prohibit their engineers from open sourcing projects or contributing to projects the organization depends on.

mcguire 9y ago

A) Actually, we have plenty of ways of measuring software engineering competence.[1][2][3] B) Weirdly, all of them are heavily process focused and none of them have much interest in the ability to write functioning code, because C) Software engineering, as a field, is built on the belief that anyone can write software if properly managed.[4] [1] http://cmmiinstitute.com/ http://cmmiinstitute.com/ [2] http://www.sei.cmu.edu/certification/ http://www.sei.cmu.edu/certification/ [3] https://www.computer.org/web/education/certifications https://www.computer.org/web/education/certifications [4] and that would make software development much cheaper.

mannykannot 9y ago
Re C): Indeed - and the belief extends to the proposition that to properly manage it, you do not need much understanding of what the detailed design and coding aspects of development actually entail, so long as you know Software Engineering.
- mcguire 9y ago
  That is an extension of the old 'managers need to know how to manage, not how to do things.'
BatFastard 9y ago
Indeed anyone can write software. However writing good, maintainable, scalable software is a totally different thing. There are so many skills (someone posted a skill matrix recently which I liked). I taught martial arts for many years, and I can honestly say that I can teach martial arts to anyone. However 98% of those learning will suck at it. They don't have the aptitude, the dedication, or pain tolerance.

mbesto 9y ago

I do technology due diligence on behalf of investors for a living, so I live and breathe this type of stuff on a weekly basis. All companies build software differently. Some have automatic deployment, some don't. Some have strong testing procedures, some don't. Just because a company doesn't use a CI, doesn't necessarily make them "worse". It's just an indifference to the indoctrination of the "SV mindset". The more important answers are not binary yes/no by rather "why aren't using a CI". Common answers are: - I'm not sure what CI is - We don't have enough unit tests to justify it - We're a small team and it doesn't really justify the effort to setup - We're working on setting up and should be live in the next 3 months You can tell a lot about engineering competency and leadership from those answers.

nudpiedo 9y ago
Not sure why you got downvoted, you are clearly speaking out of your experience working with different teams and expoisng real answers which many people could say loud every second day.
deegles 9y ago
Any advice on getting into this field? Seems like it would be pretty rewarding (mentally) work.
- mbesto 9y ago
  Honestly - I stumbled into it. I really don't have any good advice other than know the right people.
wolfgke 9y ago
My personal answer rather is: One is working on a kind of software with additional safety or security requirements, where CI would be a really bad idea. This does not contradict to the idea that methods that are rather necessary for CI, such as really high test coverage (this is typically even a requirement for such a kind of software), automatic building (can improve productivity a lot) often also make sense in such an environment.

kazinator 9y ago

Do you follow the shibboleths of my religious sect?

deleted 9y ago
[deleted]

deleted 9y ago

[deleted]

dwheeler 9y ago

If you're looking for a set of basic criteria for well-run open source software projects, please check out the CII Best Practices Badge: https://bestpractices.coreinfrastructure.org/ https://bestpractices.coreinfrastructure.org/ Full disclosure: I lead the project. Constructive comments very welcome!

jondubois 9y ago

This test completely misses what it means to be a senior (effective) engineer. The real difference between a mid-level engineer and a senior engineer is that the mid-level engineer will mechanically apply the same strategies to all projects without thinking - All the points mentioned in this article; CI, one-step deployment, daily status check-in meetings, etc, etc... are in fact not necessary for ALL projects. The quote "Those who only have a hammer tend to see every problem as a nail" is a good summary of the junior/mid-level mindset. I don't know the author, but based on the rigidity of the article, I would guess that they've only worked for big companies. I would argue that a most of these rules are only effective in the context of a very large company; in literally every other context, many of these rules are inefficient. Big companies are all about risk mitigation; they are willing to sacrifice speed and agility in exchange for stability, certainty and visibility but this is actually a luxury that only big companies can afford and should not be taken as a rule of thumb.

Profragile 9y ago

-7 thanks. I would do absolutely anything to avoid working with whoever scores very high in this test.

Silhouette 9y ago

I think I get -9 for one current project, though I'm not sure because I don't understand several of the weird ones. Then again, the client likes that project, because their customers also like it. It solves a problem for them that no-one had solved in a similar way before. I don't think we've had a single major bug reported against that part of the system by any customer in over five years, and typically that includes a multi-month lab evaluation by each customer before deployment. So, does the development process on that project suck or not? :-)

suhith 9y ago

It would be great to have some compiled data on various companies/teams and the obligations + merits that they do or do not meet.