19 ms·

Technical Interview Performance by Editor, OS, and Language

403 points by compumike 9y ago

compumike 9y ago
We’ve always allowed engineers to interview using their own preferred environment (which can be fun or challenging for us interviewers!), but now we do enough interviews every week to see which tools tend to be more successful. Also, there are some rising stars we couldn't include in this analysis, like Go: candidates writing Go during their interview with us did extraordinarily well on average, but we don't yet see enough of them to break them out of the "Other" language category.
- dasmoth 9y ago
  Do other "emerging" languages also do well?
  - deleted 9y ago
    [deleted]
- firethief 9y ago
  Once you do, will they still?
- meggar 9y ago
  What's the most popular editor for Go?
  - ramenmeal 9y ago
    I don't have any backing data, but I think vscode with the go extension is pretty popular.
    - drspacemonkey 9y ago
      Can confirm. VSCode's Go extension is really nice. I started using Atom for Go, but quickly switched over. The only thing I miss from Atom is the extension that shows inline test coverage.
      - dnasty 9y ago
        I'm not familiar with how inline Go test coverage worked in Atom but the VSCode Go extension does include it. There is an action called "Go: Toggle Test Coverage In Current Package" which I think will do the job for you.
    - compumike 9y ago
      (Article author here.) I have the data, and you're right -- a plurality of Go users were using VS Code.
  - rajathagasthya 9y ago
    Vim with vim-go plugin is really nice.
  - pk2200 9y ago
    Gogland is great, and especially easy to learn if you've used other JetBrains IDEs (IntelliJ, PyCharm, WebStorm, etc). https://www.jetbrains.com/go/ https://www.jetbrains.com/go/
  - zeveb 9y ago
    It's almost certainly not popular (IIRC vim is the most popular Go editor), but emacs with go-mode & company-go makes an extremely effective editing environment.
- sushid 9y ago
  Are there any other interesting insights? Like Java vs. Python? Do the less popular choices (e.g. Ruby/JS) do worse than the average Java/C++/Python interviewers?
- jdoliner 9y ago
  So if I write Go in vim does that mean I'm insta-hired? :P
- bsimpson 9y ago
  Curious: does Code roll-up into Visual Studio or Other in this chart?
  - compumike 9y ago
    Visual Studio
    - deleted 9y ago
      [deleted]
- dbcurtis 9y ago
  I wonder how much of that has to do with just being the kind of person that keeps up? I've always tried to at least dabble in the new, interesting languages. First, because it stretches the mind. Secondly, when I detect true progress, I want to be on top of it -- which I think is the key reason I'm still relevant as a senior individual contributor despite having started the game in the era of punch cards. People that are coming in with fluency in Go, Erlang, Haskell, etc... indicates that they are paying attention.
  - suprfnk 9y ago
    Erlang and Haskell are both around 30 years old. Could you classify that as 'keeping up'?
- tormeh 9y ago
  Go is a very small language. If the interview process requires you to write code without looking anything up, that will be easier in Go.
sevensor 9y ago
Does disseminating information like this cause a net improvement in the workforce? Will people study high-productivity editors in the hope of getting hired, only to become better programmers despite themselves? Or will this result in people learning just enough of vim or emacs to use it badly in an interview?
- echevil 9y ago
  It's a fun read :)
- kp1234321 9y ago
  Probably the latter.
- compumike 9y ago
  No, I don't think it's going to cause an improvement -- we already see plenty of individual high-productivity Eclipse users and low-productivity Vim users.
  - problems 9y ago
    I know many high productivity Vim, Emacs and IDE users - but in terms of low productivity people, IDEs definitely dominate. Really, not a jab at anyone, just that Vim and Emacs are more advanced tools and it takes more advanced users to use them with any gain. More advanced users also use IDEs for some of their nicer tooling - personally I'd kill without Resharper at work.
    - CJefferson 9y ago
      I teach, there are plenty of incompitent students using vim (and other editors of course).
- humanrebar 9y ago
  > Or will this result in people learning just enough of vim or emacs to use it badly in an interview? Hey, I use them badly in real life, too. No reason to make it sound so duplicitous.
dtran 9y ago
(Longtime emacs user but have now used sublime text for several years) Super interesting post, Mike! Not surprising that emacs/vim users tested better since use of those is likely correlated with having programmed for longer or having programmed from earlier in life before some of the other editors even existed. Would love to see it normalized by years programming. Also, insert joke about how using emacs or vim probably led to natural selection weeding out all but the most committed programmers or how the bad vim programmers didn't start their Triplebye interview because they're still trying to figure out how to toggle back to command mode to exit. :stuck_out_tongue_winking_eye: I was surprised to see such a big difference between PyCharms and Sublime. Would love to see a language breakdown for generic text editors like Sublime and Atom and compare python interviewees who used Sublime to python interviewees who used PyCharms. I also wonder if there's some confounding variable since AFAIK there's no free version of PyCharms whereas there are free versions of Sublime and Atom, so an interviewee who has a PyCharms license likely uses it in a school or professional setting.
- compumike 9y ago
  It's a reasonable hypothesis that there's probably a correlation with when someone started programming. Both editors and languages are "sticky" tool choices in that way. FYI, I think there's a free version of PyCharm (Community Edition).
  - hashkb 9y ago
    I used TextMate and then Sublime for the first half of my 10 year career. It took seeing a coworker use vim to sell me on it, and within two weeks I was a better text editor than ever before. You all know the rest of the story, but I'm one counterexample to the "how long have you been coding?" angle. I was meant to discover these editors even though I'm relatively young. (31 heh)
  - buxtehude 9y ago
    Curious if Triplebyte interviews for all of dev/ops/devops positions? Is there a higher/lower success rate in ops/devops vs pure development? For example vim is used extensively in ops/devops and emacs use there is quite rare. If there is a higher success rate in ops/devops and if vim is used more in ops/devops - could this influence the results shown? Fascinating data - thanks for sharing.
DylanDPC 9y ago
This is a shit article. There are people who use multiple IDEs as well. Also, there are people who use complex IDEs but don't know much about how to manage and just get the work done.
TomK32 9y ago
vim ~/docs/cv.md /Skills } I * vim :wq
- castis 9y ago
  you just appended your skills list with "* vim :wq" and then left the editor running.
smg 9y ago
Is this more evidence that interviews are broken in the world of GitHub profiles? There does not seem to be any correlation between the choice of editor and developer productivity when it comes to open source development. It is possible that developers who code with vim and emacs are forced to focus on low-level details like method names while developers using IDE's can focus on getting stuff done. While I do understand that this is just content marketing to drive traffic, it would be great if the following questions were handled with rigor. Is editor usage related with age of the interviewer? If there is a strong co-relation between Eclipse and older interviewees this could very well be explained by ageism. Is PyCharm ranking high in the rankings because of confounding with Python also being high?
- wutbrodo 9y ago
  There's no need to have such a defensive reaction. There's plenty of reasons a result like this could appear, and most of them don't rely on "high-quality developers prefer vim/emacs". As a tangential example, back when OSX market share started climbing a ~decade ago, it was a safe bet to say that the average OS X user was savvier than the average Windows user. This was true at a baseline regardless of what you think the "savvy" choice for an OS is: the simple fact that the sample of Windows users includes the large chunk of the population that didn't give any thought to their choice of computer (a sub-population that naturally is less savvy). This dynamic would hold true whether you think that savvy users would prefer Windows or OS X: by dint of being the default, the Windows-using population had its average savviness dragged down, a fact that has no bearing on the suitability of the OS for computer-literate users. There could easily be a similar dynamic at play here: beginner developers are going to be more likely to be on a big GUI IDE, which would drag down the measured average quality of Eclipse users. This is entirely consistent with high-quality developers preferring Eclipse to Vim, or being split 50/50 between the two, etc etc. As you point out, confounders like age and experience are also likely contributors. > It is possible that developers who code with vim and emacs are forced to focus on low-level details like method names while developers using IDE's can focus on getting stuff done. FWIW, using Vim doesn't mean using raw Vim with no plugins. I used vim my entire time at Google and I was the most productive dev on most teams I was on. I'm pretty sure this would've been impossible without plugins at least including auto-complete, jumping to definitions, etc.
jedberg 9y ago
I'd be interested to know how many PyCharm users were using standard editing mode vs Vim mode.
- otakucode 9y ago
  It's especially odd that the numbers support a strong Vim/PyCharm dominance with emacs a distant third, but Vim/emacs are the headline and area of focus. I suppose I can understand from the viewpoint that Vim/emacs are the "old/weird" editors and PyCharm is seen as a modern 'modern IDE', but I didn't expect to look at the charts and see such results from PyCharm. I happen to think PyCharm is pretty good, and use it for almost all my Python work, but didn't really think it was terribly popular overall.
rawnlq 9y ago
It is possible that eclipse users are more used to relying on their IDE for auto-completion so they can't whiteboard as effectively...
- wutbrodo 9y ago
  I hear stuff like this a lot, which seems to rely on whiteboard interviews expecting people to know method names. Do anyone but the worst interviewers actually _do_ this? That's a serious question, since my experience in this regard is pretty skewed. I've been doing whiteboard interviews for years now, including for a couple of years at Google, and the notion of ever expecting a candidate to remember a method name perfectly is beyond insane to me. I'm not OCRing their code and dumping it directly into a compiler: why the fuck would I care if they used size() or length() when checking how long a string is[1]? [1] Yes yes, I'm aware that this example isn't perfect because it could mean different things depending on the language
  - Filligree 9y ago
    I'd like to hope they don't! Interviewees, however, tend to get flustered if they can't remember the method names. It's another rock on a pile that eventually leads to nervousness collapse, something that happens far too often, so after a few dozen interviews I've started stressing up-front that no, it doesn't matter, I'm not scoring you on that. That, on its own, produced a measurable increase in average score.
    - jetsnoc 9y ago
      What do you score on?
      - Filligree 9y ago
        It all amounts to "Would I like to work with this person?", but a couple of things. Persistence, memory, ability to learn -- not trying the same thing over again if it doesn't work -- intelligence when possible, knowledge if I expect them to have it. That's pretty vague. I use the same strategy as the sibling post, ratcheting up difficulty until the candidate fails, so I usually have a lot to go on, but the exact criteria end up changing from one interview to the next depending on what the candidate did. Occasionally I end up with not enough data. I try to give my feedback in the form of a confidence interval, but if it's mediocre candidate combined with not-stellar signal quality... figuring out how to score them can take a while.
Naac 9y ago
Article title is a bit misleading. Should be "Technical Interview Performance by Editor/OS/Language"
- sctb 9y ago
  Thanks, we updated the title from “Vim and Emacs users 2x as likely to pass technical interview vs. Eclipse” a little while ago.
javajosh 9y ago
Seems like a good time to mention this oldie-but-goodie from Oliver Steele on how IDEs affect our thinking, from 2004. This blog post was a big reason I started coding with basic text editors, and I'm glad I did: http://blog.osteele.com/2004/11/ides http://blog.osteele.com/2004/11/ides
shmerl 9y ago
I'm surprised macOS is more used than Linux among developers.
- deathanatos 9y ago
  The last three companies I've worked for didn't offer Linux as a choice, so I would say it is going to be underrepresented. The choice was always MacBook or some laptop running Windows, and between those, I'll take the *nix-like OS. But it isn't my preference, and I'd be a lot happier on a Linux laptop. (For potentially more reasons than just the OS, too.) (Now, I'm not entirely clear on how Triplebyte is getting their data. Do they ask explicitly, or pull it from the UA header? People could, I suppose, be doing it from their work laptop. Or, most people just don't care to customize that much, and just use macOS because it is good enough for them.)
  - shmerl 9y ago
    I'm glad my current place offers Linux as an option.
- komali2 9y ago
  I'm not. I am not a fan of apple at all (for various arbitrary reasons, and a couple reasons I think I could make a valiant argument about), but I have to admit a macbook pro out of the box does "just work." I throw on chrome and my editor of choice, maybe xcode or whatever if I need it, set up my ssh keys, install git, swap ctrl/cmd, done. I can plug it into an external monitor and unplug it without is losing workspace/desktop context, I know I'm getting the best battery life, the resolution is working straight out of the box in all applications, no random slowdowns, etc. Meanwhile every time I try to put Linux of any flavor onto a macbook or a lenovo carbon or whatever, something gets fucky. Video drivers don't work, no webcam driver exists, the mouse glitches, the tilde key doesn't work, plugging into/out of an external monitor crashes it or causes all my windows to collapse into a single workspace, same issue for if the computer goes to sleep while connected to an external monitor, all sorts of weird things causes it to not sleep on laptop lid close and nukes my battery, battery life will drain excessively, arbitrarily, chrome will randomly cause the computer to slow WAY down, etc. Ok, that was a rant, but TLDR - I hate macs but coding on one just removes all the distractions from programming and let me "just do my job." (I am experimenting with windows as well right now because I love the Surface platform as a whiteboard and prototyper, so far WSL is great and I actually prefer it over linux because all the OS stuff "just works." Lots of weirdness around WSL still though)
tapatio 9y ago
This is most likely because engineers that use vi/emacs have to memorize language syntax and APIs as vi/emacs don't have code completion like most IDEs (at least not by default). And having that stuff memorized means you'll do better on a coding interview.
- xena 9y ago
  Fuzzy-matching autocomplete for vim: https://github.com/Valloric/YouCompleteMe https://github.com/Valloric/YouCompleteMe Fuzzy-matching autocomplete for emacs: https://company-mode.github.io https://company-mode.github.io If you are at all interested in trying something with all the bells and whistles turned on for you, check out http://spacemacs.org http://spacemacs.org
- michaelmrose 9y ago
  This is complete nonsense. It's completely trivial to get code completion. Its wholely unrealistic to suppose that someone would spend hours to days learning emacs but not spend 1 minute adding a completion plugin.
  - sidlls 9y ago
    I don't use any completion plugin with my Emacs. Not coincidentally I'm generally a better reference for our programs' APIs and organization than the documentation or the other developers who use things like PyCharm and Sublime. For me not using completion works to keep me disciplined about the structure of code I work on. It also can make me vicious about arity and composability in code reviews. Fortunately most of those details are taken care of by design prior to implementation, so it's rarely an issue.
    - michaelmrose 9y ago
      I think not using code completion might have more to do with you having an excellent memory and it might not be fantastic advice for everyone but I would be interested if there were stats on usage of features like code completion in emacs.
    - 9y ago
tritium 9y ago
And yet, it's still all just plain text editing. Clearly, one kind of answer is more fashionable than the other.
look_lookatme 9y ago
I would not extrapolate too much from our broken system of technical interviews.
jawns 9y ago
Sample size is never directly mentioned, but: > At Triplebyte we evaluate hundreds of engineers every week. > I pulled the editor / language data for every Triplebyte interview over the last year. Based on that, if we suppose "hundreds" means "low hundreds," this suggests a sample size of 5K-15K. That's not a bad sample size for the total population. But if 35% use Python and only 5% use Ruby, then you're comparing a sample of 1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby data is more shaky than the Python data, and thus it's not a fair comparison. Same goes for a lot of the other metrics where the sample sizes vary. ~ Disclaimer: I'm no statistician (but I play one on Correlated.org).
- tedsanders 9y ago
  Be careful equating 'evaluate' with 'interview', when TripleByte has incentive to obfuscate. I suspect hundreds of engineers may apply through Triplebyte each week, but they may interview only a small fraction of those. If true, this makes your point about small sample size even more important. I was disappointed their charts didn't have confidence intervals or count numbers. Makes it harder to guess at how the numbers generalize.
- jayant123 9y ago
  Also, my guess is that the success rate for interviews would be fairly low, making any comparison of the success rates for different languages/editors even more statistically shaky. They should definitely publish the total sample size and success percentage for these results to be meaningful. Example: Say they hired 5 out of a 100 applicants who used Python, and 2 out of 20 applicants who used Ruby. That's a 5% success rate vs a 10% success rate, a 100% increase. I'd hardly call that a significant result though. Disclaimer: I'm no statistician either
minikites 9y ago
As so many Hacker News posters are fond of saying, correlation isn't sufficient to imply causation. Or perhaps their interview questions are designed in a way that biases a particular development environment (positively or negatively).
- danjoc 9y ago
  Or perhaps there's some racial bias and a certain ethnicity tends to be associated with Java and Eclipse.
- praneshp 9y ago
  > correlation isn't sufficient to imply causation. The article calls this out in the first paragraph.
CoreXtreme 9y ago
I've not read it! But experienced folks will be hired more often than inexperienced ones. And lots of old programmer use vim/emacs and other CLI editors. So, it's not surprisingly. Do they've data adjusted for programming experience?
hellofunk 9y ago
I started coding professional on Xcode. Then I tried a couple of other IDEs until my work in lisp required I give emacs a try. I eventually went back to other languages, mostly C++ and Python, but still use emacs. It really spoils you for text/code editing forever. Once you get passed the learning curve, every other effort to edit code or text in other programs seems ironically so archaic. What? I can't split the buffer quickly and get another one? What, I can't quickly bounce back to my previous editing window configuration, then bounce forward again? What, there is no visual undo tree? The list goes on and on. I now use the Vim emulation inside emacs which has been the killer feature in emacs to further boost productivity. And, last year I passed several rounds of technical interviews with one of the top 5 companies in SV, so perhaps I support the statistics presented here (though I did not get to use my chosen editor for the interviews, so they did not know I am an emacs user).
- agumonkey 9y ago
  I colleged on Eclipse. But I always had a thing for emacs (it has some parsing abilities oob, it's not just coloring and lexing). Anyway the day I finally clicked on the extensibility of emacs. Which was as trivial as defining a function, then defining a menu (which is duh.. a list). And have the GUI [1] updated I lost the remaining bit of love I had for Eclipse. I got 2 or 3 years of Eclipse JDT/PDT, all the talks about the power of plugins. It was nothing but pain. Then that old emacs gets me the same thing like it's breakfast. It hit me too hard. Lastly, I revived a win95 box, with Turbo Pascal. Another old and tiny blast from the past, and even though the 600KB IDE/compiler was amazing in most and so many regards. The fact that I was stuck with the default text editing features (no paragraph selection etc etc) made me feel what Emacs' freedom meant to me even on the tiniest instances. [1] note that I'm a keyboard fetishist, so I don't use 99% of Emacs GUI and knew most keybindings, I was just trying for the sake of learning elisp and emacs architecture.
todd8 9y ago
Fascinating, but what does it mean. I can dream up explainations, now after the fact, for this and the other interesting results in the article, but I wonder what the real reasons are? For example, Eclipse is the standard University Java course editor. Users that have switched to IntelliJ have probably had more time away from University and hence more experience. Is this the reason, or is it a "just so story"?
- stupidcar 9y ago
  Perhaps, but a lot of teaching is done with Python nowadays, and Python has a positive pass-rate, and actually tops the on-site success rate. So it doesn't just seem to be a case of beginners getting rejected a lot. Based on my anecdotal evidence of interviewing Java devs, I'd suggest the problem is the opposite. While there are some excellent Java coders out there, there are also an awful lot of "experienced" devs, with fifteen years of experience and salary expectations to match, whose technical ability proves to be pretty miserable.
deleted 9y ago
[deleted]
otakucode 9y ago
I think the headline kinda buries what I take to be the lesson from the data... companies don't like Java devs. Which is understandable, I imagine they find dealing with their ResumeAdapterFactoryInterfaceFactoryAdapterInterface needlessly gross.
- optimusclimb 9y ago
  That's such a dead and boring meme. Honestly the attacks I always see on Java (such as this one) always come off as completely naive. The amount of open source big data projects, and large profits and companies relying on Java technology should clue you in that this is unfounded. Hell, half of the modern smartphone market is all about Java. I don't think companies dislike Java devs. I'd also take an article from a company trying their hardest to commoditize engineers and turn them in to test scores with a grain of salt.
coldcode 9y ago
I have been coding for 36 years and currently do Swift/XCode. Used plain text editors back in 83/84 time frame but have using some kind of IDE since Turbo Pascal 1.0 and have zero desire to use either vim or emacs. I wouldn't hold a lot to the idea that old programmers use them more than young. People should use whatever they feel makes them more productive.
- jhgb 9y ago
  But (Gnu) Emacs is not a plain text editor. It can edit formatted texts, embed pictures and widgets etc. While XCode's editor, even if I haven't seen it in practice, most probably actually is. So it can't be the "plain text editor" aspect that is the differentiator in favor of XCode here.
weeksie 9y ago
If you're an emacs user there's nothing worse than doing an on-site interview where someone wants you to pair program using their editor. Nothing works right and you spend half the time flipper-handing it over the keys trying to remember how normal editors work. Then again, it's rare that I interview anymore. I've done maybe two of those kinds of (pair programming, someone else's rig) interviews in the last ten years and neither was for a job that I really wanted after spending time with the devs conducting the interviews. You want me to consult with you? Let me talk to your biz people to see if I'm the right person to solve your problems, and let the technical folk go over code I've written and we can deep dive on that if there is any question about my chops. Otherwise, fuck off. I'm too old to play "dance monkey".
- psynapse 9y ago
  "Let me talk to your biz people to see if I'm the right person to solve your problems, and let the technical folk go over code I've written and we can deep dive on that if there is any question about my chops." I feel you here. With becoming longer of tooth (in the trade) comes an understanding of how to contribute value via the multitude of avenues that are not just grinding out lines at the keyboard. I am surprised over and over at the incandescently bright young people who can wax lyrical about monad transformers, but to whom it does not occur to model a domain properly before cutting code, or who lack basic schema design and/or SQL knowledge. I am also in the boat of having vanishingly few technical interviews in recent years, but on a good deal of those occasions it has been obvious that the interviewer is just so not the person I should be talking to.
  - quickthrower2 9y ago
    Modelling a domain is very underrated by companies as well as individuals and it's a shame because we end up with code bases full of mixed purposes language and confused naming that makes it harder to work on legacy. Monad Transformers FWIW are bread and butter in Haskell. So it a bit like going on about dependency injection. Not necessarily a sign of being stuck up or head in the clouds.
    - 9y ago
fcbrooklyn 9y ago
I'd be interested to know how their prescreening works. Sometimes those things are timed, and I can imagine a really short timeframe, say 30 minutes or so, might make the specific editor more relevant. For example, I absolutely hate Eclipse. Back when I was writing Java, I reached into my own pocket to buy IntelliJ because I was at least 50% more productive with it. I know there are people out there who dig Eclipse, but even if it's your editor of choice, it's more appropriate for dealing with a big, long-lived project. Similarly, if they had some existing project you are required to work with, full on IDEs like Eclipse might require a lot more fiddling to work properly, compared to straight up editors like vim/emacs/etc.
jorgemf 9y ago
Flame war ahead... I want you to bear in mind that this depends highly in the number of open positions per language/technology and the number of candidates per position. Does it mean I suck if I do Java in Eclipse compare to someone who does Ruby in Vim? Maybe they have 1 java position for 1000 java developers and 100 Ruby positions for 500 Ruby developers. So basically this article says nothing. It needs more information to be relevant.
- bitwize 9y ago
  That stuck out to me too. The "ideal coder" uses a Mac, Ruby, and vim -- pretty much a snapshot of the stereotypical Bay Area startup hipster. It could be that that's what they -- or their clients -- are selecting for.
- compumike 9y ago
  (Article author here -- I'm one of the interviewers here as well.) Our technical interview is really designed to look for your strengths, such as being productive while writing clean, correct code in any language/environment of your choice. If you do well at that (or excel in other sections of the interview such as system design or knowledge questions), we're going to work with you as a candidate, regardless of the language you choose to interview in.
  - jorgemf 9y ago
    Still is irrelevant. If you have 1 java position and 10000 Ruby positions, you are going to interview more Ruby developers, your data is going to match more closely the reality. For the 1 java position you are going to be more picky and discard more candidates. But even with the same number of candidates and positions your test are going to have bias, how do you know the java test is as difficult as the ruby test, it could be more or less difficult. You can do a proper analysis fixing some of the variables. For example you can fix the language and compare how the different editors affect the selection process. But still you have to show other variables as the years of experience in the language. And finally you have outlayers like who did an interview with you but was discarded because of living in a different country. But I guess I might have enter into the statistics.
pfarnsworth 9y ago
I find it kind of funny and rather llazy that they don't dig into deeper as to why there's such disparity between programming languages. To me, it's probably that the questions are geared towards Python/Ruby, rather than C++. For example, parsing a CSV is one line of code in Python/Ruby by default, but if you're supposed to do this in C++, that alone can take 10 mins to write and test, etc. If they really want to make their test/evaluation fair, and if they truly want to be inclusive in the languages, they should figure out why there is such disparity, it's probably with the questions they ask rather than the candidates. To me, looking at these numbers, it just means that were I ever to interview with them (I never will because I think they offer no benefits whatsoever, since I don't want to interview with them, and then with the external companies again when I can just go with the external companies the first time), I would just choose Python or Ruby to interview in, since the tests appear more geared to those languages.
WhitneyLand 9y ago
Programmer interviews are broken. Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code. Technical hiring processes harm both excellent candidates who don't interview well and companies trying to hire good programmers. Agreed. Great start. [make an] investment by going through our interview process, afterwards we can fast track you through to final interviews...O(n) process into an O(1) process Makes sense. Use of complexity notation is transparently gratuitous, but I liked it, because it shows the intention to understand and connect. Traveling the world most people appreciate an honest attempt to speak to them in their language even if you're not fluent. You'll have to take a coding quiz with questions in several different languages, as well as some design questions. The coding questions involve reading blocks of code, and answering questions about them. Intuitively I hate the idea because I'm skeptical of how effective it is. Do you have non-anecdotal data to show how well your quiz correlates to specific goals? After the quiz, you may be asked to complete a coding challenge... What triggers that requirement? Also same question as previous question about correlation/validity. Then, you will move on to our technical interview... What do the tech interviews consist of? They can vary wildly across companies, interviewers, and formats. ...work history [is] meaningful but relying solely on [it] results in missing good programmers This is a show stopper for me. If a company doesn't acknowledge that what people have accomplished with previous contributions, projects, development work is not just meaningful, but extremely important and a top criterion to find great people, then I don't believe they have the correct perspective. But hey, I can be wrong about things, I'll try to be open minded. Point me to solid data showing tests and quizzes are a better predictor than accomplishments and I'll have a look.
- mal808 9y ago
  Regarding the work history thing, I think the point is missing good programmers who have a patchy work history. I, for example am 42, but I've only been a programmer for 3 years. Someone looking purely at my work history probably wouldn't have bothered to interview me for the first couple of years. Fortunately for me that wasn't the case.
  - WhitneyLand 9y ago
kybernetikos 9y ago
When it comes to a choice of programming language, ultimately a right tool for the right job mentality is good, and for the kinds of tasks that commonly come up in interview situations, C++ (or even Java) is not usually the right tool. This of course doesn't mean that for real life situations, they're never the right tool, but if someone chooses C++ to solve a simple tree inversion or sliding window problem in an interview situation it probably means that they don't have sufficient familiarity with any language more appropriate to the situation.
- dabei 9y ago
  C++ in the right hand is very succinct and expressive for solving algorithm problems. If someone claim c++ expertise but switch to their secondary language for some questions I'd think twice about hiring them.
  - kybernetikos 9y ago
    This is true - there are some people who can make C++ sing even for these kinds of problems, however most people who attempt to use C++ in an interview situation to solve toy interview problems do poorly. For most people, it takes them longer to solve the problem, to find and correct bugs and to provide tests that prove that they've solved the problem correctly than someone solving the same problem in a different language. I've noticed this in my own interviews, and this article mentions it too. It seems to me that the most natural explanations for this are either that the pool of C++ programmers has more bad programmers in it than other languages (which I don't believe) or the language and tools themselves are not as well suited to solving interview questions in interview situations. As to penalising people who switch languages for problems, if you give guidance that you want to see a solution in a particular language, that's fair enough, in any other case it seems unfair and misguided. Perhaps you don't really agree with 'right tool for the right job' when it comes to languages?
- drharby 9y ago
  But google told me that my language choice won't impact their hiring decision :(
c3534l 9y ago
The low score of C++ and high score of Python in pass rate made me think for a moment. How are they determining pass or fail rate? Because if they're failing people because they forgot a semicolon or something trivial like that, then this ceases to be a test of interview performance and more a test of how many inconsequential mistakes does the average person make. So for instance, Vim doesn't provide much help when you mess up. If you type a variable name that doesn't exist, that's your fault (I'm sure with plugins you can get more). So do Vim users do better because they're nerdier, don't mind a steep learning curve, and like working in a terminal, or is it because Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"?
- xiphias 9y ago
  In my experience the new ,,cool'' startups prepare interview questions in python/ruby/..., give some time, and don't measure the time spent by the program. In this setup writing the same code in C++ (including the right STL headers) is a pain in the a. I remember that when I selected C++ as my language for interviewing, I was strongly punished, because my interviewer didn't understand why it takes 10 minutes to do something relatively simple (that is just a few line of code in a higher level language). After that I switched to using Ruby in my interviews, and passed easily.
- rch 9y ago
  Their entire sample seems to be composed of people who are willing to click on a green button and "take the quiz" as part of a job search.
- thomastjeffery 9y ago
  > If you type a variable name that doesn't exist [in vim] Start typing a work and press C-x C-[nolk] * n for a word in open buffers * o for "omnicomplete" which is quite similar to "intellisense" * l for the entire line in open buffers * k for the word from a dictionary There are several others, but those are the main ones. As soon as you start a completion, you can use C-p and C-n to go up and down the list respectively. > Vim doesn't provide much help when you mess up Vim is designed for editing text. I don't know if a better editor (yet) suited to editing, rather than writing. > Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"? It can, so long as you have it set up to. Vim even has a spell checker.
didibus 9y ago
This is missing data about the job role. Are python devs hired as web dev, and c++ as 3D engine programmers? Does the triplebyte hiring process not take into account experience and role? Don't they expect more for certain roles and a given experience?
- dllu 9y ago
  It seems Triplebyte just tests everyone on web technologies regardless of role or experience. It makes sense that C++ engineers are less familiar with large scale web apps, but are valuable hires in the industry for robotics, finance, game engines, etc.
  - compumike 9y ago
    (Article author here.) I'm one of the interviewers. If I had to guess, maybe a third of our generalist-track interview is web-related. I've personally interviewed many engineers who did not know the first thing about any web stuff, but were particularly strong on other sections, so we accepted them. You really don't have to be great at every section to pass the Triplebyte interview; you just have to show your personal areas of strength. And we give you lots of different opportunities to do that. Of course, we will then match you with appropriate roles and avoid the ones that require web-specific skills. FYI, we already have robotics, financial, and game companies on our platform!
Rantenki 9y ago
This whole article reads like an celebration of confirmation bias, which could be summarized as: "The people that pass our interviews tend to be the ones that use the tools that we prefer." I don't think there is any value in looking at this blog post and making any conclusions about which editor and/or scripting language to learn next. You might go interview at a .net shop and get the opposite result from the one that you wanted. Disclaimer: I personally match their preferred technologies very closely, but work with some really smart people that write C#/F# in VSCode.
- davesque 9y ago
  These were my thoughts exactly. Other things can also come into play. For example, what if certain technologies demand higher levels of skill? Then the interview pass rate might naturally be lower because the bar is higher. There could be countless ways that these numbers actually mean something different than what is implied.
- cableshaft 9y ago
  There are a lot of really smart C# programmers out there... but there are a lot of subpar C# developers out there too, especially outside of Silicon Valley. I've had a boss who brought in at least 60 people in (while I was there) for an unsupervised (unless you had a question) 60-90 minute test, where he'd leave them alone, let them google, etc, but would sit down and discuss what they did and why afterwards, and the amount of people who couldn't get past the first FizzBuzz question was pretty high. Those that got past that, the number who didn't know the basics, like what an Interface was or how they worked (or how you're not supposed to write implementation code directly into them) was higher still. And these are people who say they have 5-15 years of direct experience on their resumes. I'm not a huge fan of tests and I do tend to struggle myself, but not that badly, and not when unsupervised.
  - kevindqc 9y ago
    I never understood and that can happen. What have they been doing for 5-15 years? I can understand performance anxiety being a factor.. but 60-90 minutes and still can't answer FizzBuzz?
wazoox 9y ago
In my experience with many interns in the past few years, most of them came using Sublime Text, in the unpaid nagware version; however all of the most promising and intelligent ones moved forward to other editors, most of them going to Vim.
mulmen 9y ago
This screams of lying with statistics. There are no interesting conclusions drawn here, just some correlation that is at risk of being based on selection bias. This is just fodder for the endless tech variants of ford-vs-chevy debates. The elephant in the room is that we don't even know if technical interview performance is a good selector. I'd be a lot more interested if this data included long-term results of actual job performance like employee and manager satisfaction, time in role, time to promotion, etc.
seizethecheese 9y ago
This just makes me wonder what factors are correlated with Eclipse editor usage. Perhaps there's a sub-optimal channel to the developer career path where people are taught Java on Eclipse. Does anyone know?
walshemj 9y ago
the OS break down was interesting as I would bet that most of the jobs are using Linux's to run the code on the massive over representation of OSX is odd. Buying a more expensive Mac to use just as a terminal just seams an odd choice.
mizzao 9y ago
Aside from the correlation !=> causation issues, there's a ton of endogeneity and selection bias in this data. The initial programming quiz has code snippets only in Javascript, Ruby, and C. The interviewers are usually more familiar with more modern technologies. Indeed "enterprise" (C# or Java) developers tend to do poorly at TripleByte, and Java is highly correlated with Eclipse usage. TripleByte also recruits primarily for startups which tend to use certain technologies. I'd say the final stage offer data is so biased by the selection process along the way that it's almost impossible to draw any conclusions from it.
arjie 9y ago
Interesting. Triplebyte is successful if the candidates they recommend are candidates the companies hire. That means that by the time a company sees a Triplebyte candidate, the candidate's language/editor choice should have no signal left in it if Triplebyte has perfect performance. Quite cool that they're willing to share fairly transparent performance metrics with us.
Karrot_Kream 9y ago
I'd love to know whether these percentages were normalized by the number of people in each bucket. i.e. we should calculate P(pass | <emacs>) or P(pass | <c#>) and see how that compares to P(pass) as a whole.
- compumike 9y ago
  Yes, that's already what we're showing. On the "Triplebyte Technical Interview Pass Rate by Editor" chart, Vim is shown as +38%. This means that: P(pass | vim) / P(pass) = 1.38
  - deleted 9y ago
    [deleted]
btilly 9y ago
I would love to see error bars on all of the graphs. Since we're estimating a pass/fail ratio, any of the confidence intervals in https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval https://en.wikipedia.org/wiki/Binomial_proportion_confidence... could be used. (Jeffreys interval may be a good default.)
dsacco 9y ago
With respect to languages: is it the case that Ruby programmers do better, or are companies hiring Ruby programmers more lenient? (Ruby being an example; Python also applies here, whereas the converse would be a hypothesis that C++ programmers are not less skilled but rather are judged more harshly in interviews).
luord 9y ago
> if you walk into any tech company in San Francisco you’re inevitably going to count far more than 60% MacBooks This surprised me. It might be because I'm not from the USA but in every company I've worked for on-site (including the current one), the overwhelming majority of developers used and preferred linux. That aside, I'm glad python does well.
caconym_ 9y ago
Interviewing in Python for my semi-dream job is one of the best decisions I ever made (on par with switching from Emacs to Vim). I don't even know the language that well, I'm an intermediate user at best, but the the reduction in cognitive load from how straightforward and ergonomic it is is crazy. It was also nice that they had an editor available with serviceable Vim bindings. I think it was a big improvement over coding on a whiteboard.
KirinDave 9y ago
And here I thought we had hit peak senseless-pattern-matching with the tabs-vs-spaces dataset. I was hoping for a quick section introspecting on correlation vs causation and what this says about their process as opposed to a semi-serious presentation of predictive power. I can only assume a total lack of rigor and process if this is what they choose to talk about publicly.
tarboreus 9y ago
The post is an enjoyable one, but makes a couple of assumptions, one being that experience and coding ability are what get you hired. Sure, they help, and they're a prerequisite, but showing up wearing the right clothes and having the right attitude might actually matter more. People who use OSX are more likely to be the kind of dev with the tidy beard and the square glasses and the decent shirt. Devs using Emacs in Linux are more likely to be the aneruotypical who doesn't look their interviewer in the eye. Both are more likely to be good coders, though, since you don't get taught Vim and Emacs in school, you have to pick it up through a non-school network or through self study and research. Eclipse is the kind of thing you use because your professor taught you Java and you never did any coding that wasn't for class. Sublime is the kind of thing you use if you just don't really care about your editor one way or the other. I'd be curious to know what kind of shop they are at TripleByte. Something tells me they might be Ruby coders using OSX all the time. Like attracts like, especially in interviews.
deleted 9y ago
[deleted]
linguistbreaker 9y ago
This "data" is not very meaningful. It says much more about Triplebyte than about developers, technologies or interviews.
brisance 9y ago
Seems like machine learning could be used to provide greater insight.
AzzieElbab 9y ago
Someone ought to put together performance charts by Zodiac signs

403 points by compumike 9y ago

We’ve always allowed engineers to interview using their own preferred environment (which can be fun or challenging for us interviewers!), but now we do enough interviews every week to see which tools tend to be more successful. Also, there are some rising stars we couldn't include in this analysis, like Go: candidates writing Go during their interview with us did extraordinarily well on average, but we don't yet see enough of them to break them out of the "Other" language category.

dasmoth 9y ago
Do other "emerging" languages also do well?
- deleted 9y ago
  [deleted]
firethief 9y ago
Once you do, will they still?
meggar 9y ago
What's the most popular editor for Go?
- ramenmeal 9y ago
  I don't have any backing data, but I think vscode with the go extension is pretty popular.
  - drspacemonkey 9y ago
    Can confirm. VSCode's Go extension is really nice. I started using Atom for Go, but quickly switched over. The only thing I miss from Atom is the extension that shows inline test coverage.
    - dnasty 9y ago
      I'm not familiar with how inline Go test coverage worked in Atom but the VSCode Go extension does include it. There is an action called "Go: Toggle Test Coverage In Current Package" which I think will do the job for you.
  - compumike 9y ago
    (Article author here.) I have the data, and you're right -- a plurality of Go users were using VS Code.
- rajathagasthya 9y ago
  Vim with vim-go plugin is really nice.
- pk2200 9y ago
  Gogland is great, and especially easy to learn if you've used other JetBrains IDEs (IntelliJ, PyCharm, WebStorm, etc). https://www.jetbrains.com/go/ https://www.jetbrains.com/go/
- zeveb 9y ago
  It's almost certainly not popular (IIRC vim is the most popular Go editor), but emacs with go-mode & company-go makes an extremely effective editing environment.
sushid 9y ago
Are there any other interesting insights? Like Java vs. Python? Do the less popular choices (e.g. Ruby/JS) do worse than the average Java/C++/Python interviewers?
jdoliner 9y ago
So if I write Go in vim does that mean I'm insta-hired? :P
bsimpson 9y ago
Curious: does Code roll-up into Visual Studio or Other in this chart?
- compumike 9y ago
  Visual Studio
  - deleted 9y ago
    [deleted]
dbcurtis 9y ago
I wonder how much of that has to do with just being the kind of person that keeps up? I've always tried to at least dabble in the new, interesting languages. First, because it stretches the mind. Secondly, when I detect true progress, I want to be on top of it -- which I think is the key reason I'm still relevant as a senior individual contributor despite having started the game in the era of punch cards. People that are coming in with fluency in Go, Erlang, Haskell, etc... indicates that they are paying attention.
- suprfnk 9y ago
  Erlang and Haskell are both around 30 years old. Could you classify that as 'keeping up'?
tormeh 9y ago
Go is a very small language. If the interview process requires you to write code without looking anything up, that will be easier in Go.

sevensor 9y ago

Does disseminating information like this cause a net improvement in the workforce? Will people study high-productivity editors in the hope of getting hired, only to become better programmers despite themselves? Or will this result in people learning just enough of vim or emacs to use it badly in an interview?

echevil 9y ago
It's a fun read :)
kp1234321 9y ago
Probably the latter.
compumike 9y ago
No, I don't think it's going to cause an improvement -- we already see plenty of individual high-productivity Eclipse users and low-productivity Vim users.
- problems 9y ago
  I know many high productivity Vim, Emacs and IDE users - but in terms of low productivity people, IDEs definitely dominate. Really, not a jab at anyone, just that Vim and Emacs are more advanced tools and it takes more advanced users to use them with any gain. More advanced users also use IDEs for some of their nicer tooling - personally I'd kill without Resharper at work.
  - CJefferson 9y ago
    I teach, there are plenty of incompitent students using vim (and other editors of course).
humanrebar 9y ago
> Or will this result in people learning just enough of vim or emacs to use it badly in an interview? Hey, I use them badly in real life, too. No reason to make it sound so duplicitous.

dtran 9y ago

(Longtime emacs user but have now used sublime text for several years) Super interesting post, Mike! Not surprising that emacs/vim users tested better since use of those is likely correlated with having programmed for longer or having programmed from earlier in life before some of the other editors even existed. Would love to see it normalized by years programming. Also, insert joke about how using emacs or vim probably led to natural selection weeding out all but the most committed programmers or how the bad vim programmers didn't start their Triplebye interview because they're still trying to figure out how to toggle back to command mode to exit. :stuck_out_tongue_winking_eye: I was surprised to see such a big difference between PyCharms and Sublime. Would love to see a language breakdown for generic text editors like Sublime and Atom and compare python interviewees who used Sublime to python interviewees who used PyCharms. I also wonder if there's some confounding variable since AFAIK there's no free version of PyCharms whereas there are free versions of Sublime and Atom, so an interviewee who has a PyCharms license likely uses it in a school or professional setting.

compumike 9y ago
It's a reasonable hypothesis that there's probably a correlation with when someone started programming. Both editors and languages are "sticky" tool choices in that way. FYI, I think there's a free version of PyCharm (Community Edition).
- hashkb 9y ago
  I used TextMate and then Sublime for the first half of my 10 year career. It took seeing a coworker use vim to sell me on it, and within two weeks I was a better text editor than ever before. You all know the rest of the story, but I'm one counterexample to the "how long have you been coding?" angle. I was meant to discover these editors even though I'm relatively young. (31 heh)
- buxtehude 9y ago
  Curious if Triplebyte interviews for all of dev/ops/devops positions? Is there a higher/lower success rate in ops/devops vs pure development? For example vim is used extensively in ops/devops and emacs use there is quite rare. If there is a higher success rate in ops/devops and if vim is used more in ops/devops - could this influence the results shown? Fascinating data - thanks for sharing.

DylanDPC 9y ago

This is a shit article. There are people who use multiple IDEs as well. Also, there are people who use complex IDEs but don't know much about how to manage and just get the work done.

TomK32 9y ago

vim ~/docs/cv.md /Skills } I * vim :wq

castis 9y ago
you just appended your skills list with "* vim :wq" and then left the editor running.

smg 9y ago

Is this more evidence that interviews are broken in the world of GitHub profiles? There does not seem to be any correlation between the choice of editor and developer productivity when it comes to open source development. It is possible that developers who code with vim and emacs are forced to focus on low-level details like method names while developers using IDE's can focus on getting stuff done. While I do understand that this is just content marketing to drive traffic, it would be great if the following questions were handled with rigor. Is editor usage related with age of the interviewer? If there is a strong co-relation between Eclipse and older interviewees this could very well be explained by ageism. Is PyCharm ranking high in the rankings because of confounding with Python also being high?

wutbrodo 9y ago
There's no need to have such a defensive reaction. There's plenty of reasons a result like this could appear, and most of them don't rely on "high-quality developers prefer vim/emacs". As a tangential example, back when OSX market share started climbing a ~decade ago, it was a safe bet to say that the average OS X user was savvier than the average Windows user. This was true at a baseline regardless of what you think the "savvy" choice for an OS is: the simple fact that the sample of Windows users includes the large chunk of the population that didn't give any thought to their choice of computer (a sub-population that naturally is less savvy). This dynamic would hold true whether you think that savvy users would prefer Windows or OS X: by dint of being the default, the Windows-using population had its average savviness dragged down, a fact that has no bearing on the suitability of the OS for computer-literate users. There could easily be a similar dynamic at play here: beginner developers are going to be more likely to be on a big GUI IDE, which would drag down the measured average quality of Eclipse users. This is entirely consistent with high-quality developers preferring Eclipse to Vim, or being split 50/50 between the two, etc etc. As you point out, confounders like age and experience are also likely contributors. > It is possible that developers who code with vim and emacs are forced to focus on low-level details like method names while developers using IDE's can focus on getting stuff done. FWIW, using Vim doesn't mean using raw Vim with no plugins. I used vim my entire time at Google and I was the most productive dev on most teams I was on. I'm pretty sure this would've been impossible without plugins at least including auto-complete, jumping to definitions, etc.

jedberg 9y ago

I'd be interested to know how many PyCharm users were using standard editing mode vs Vim mode.

otakucode 9y ago
It's especially odd that the numbers support a strong Vim/PyCharm dominance with emacs a distant third, but Vim/emacs are the headline and area of focus. I suppose I can understand from the viewpoint that Vim/emacs are the "old/weird" editors and PyCharm is seen as a modern 'modern IDE', but I didn't expect to look at the charts and see such results from PyCharm. I happen to think PyCharm is pretty good, and use it for almost all my Python work, but didn't really think it was terribly popular overall.

rawnlq 9y ago

It is possible that eclipse users are more used to relying on their IDE for auto-completion so they can't whiteboard as effectively...

wutbrodo 9y ago
I hear stuff like this a lot, which seems to rely on whiteboard interviews expecting people to know method names. Do anyone but the worst interviewers actually _do_ this? That's a serious question, since my experience in this regard is pretty skewed. I've been doing whiteboard interviews for years now, including for a couple of years at Google, and the notion of ever expecting a candidate to remember a method name perfectly is beyond insane to me. I'm not OCRing their code and dumping it directly into a compiler: why the fuck would I care if they used size() or length() when checking how long a string is[1]? [1] Yes yes, I'm aware that this example isn't perfect because it could mean different things depending on the language
- Filligree 9y ago
  I'd like to hope they don't! Interviewees, however, tend to get flustered if they can't remember the method names. It's another rock on a pile that eventually leads to nervousness collapse, something that happens far too often, so after a few dozen interviews I've started stressing up-front that no, it doesn't matter, I'm not scoring you on that. That, on its own, produced a measurable increase in average score.
  - jetsnoc 9y ago
    What do you score on?
    - Filligree 9y ago
      It all amounts to "Would I like to work with this person?", but a couple of things. Persistence, memory, ability to learn -- not trying the same thing over again if it doesn't work -- intelligence when possible, knowledge if I expect them to have it. That's pretty vague. I use the same strategy as the sibling post, ratcheting up difficulty until the candidate fails, so I usually have a lot to go on, but the exact criteria end up changing from one interview to the next depending on what the candidate did. Occasionally I end up with not enough data. I try to give my feedback in the form of a confidence interval, but if it's mediocre candidate combined with not-stellar signal quality... figuring out how to score them can take a while.

Naac 9y ago

Article title is a bit misleading. Should be "Technical Interview Performance by Editor/OS/Language"

sctb 9y ago
Thanks, we updated the title from “Vim and Emacs users 2x as likely to pass technical interview vs. Eclipse” a little while ago.

javajosh 9y ago

Seems like a good time to mention this oldie-but-goodie from Oliver Steele on how IDEs affect our thinking, from 2004. This blog post was a big reason I started coding with basic text editors, and I'm glad I did: http://blog.osteele.com/2004/11/ides http://blog.osteele.com/2004/11/ides

shmerl 9y ago

I'm surprised macOS is more used than Linux among developers.

deathanatos 9y ago
The last three companies I've worked for didn't offer Linux as a choice, so I would say it is going to be underrepresented. The choice was always MacBook or some laptop running Windows, and between those, I'll take the *nix-like OS. But it isn't my preference, and I'd be a lot happier on a Linux laptop. (For potentially more reasons than just the OS, too.) (Now, I'm not entirely clear on how Triplebyte is getting their data. Do they ask explicitly, or pull it from the UA header? People could, I suppose, be doing it from their work laptop. Or, most people just don't care to customize that much, and just use macOS because it is good enough for them.)
- shmerl 9y ago
  I'm glad my current place offers Linux as an option.
komali2 9y ago
I'm not. I am not a fan of apple at all (for various arbitrary reasons, and a couple reasons I think I could make a valiant argument about), but I have to admit a macbook pro out of the box does "just work." I throw on chrome and my editor of choice, maybe xcode or whatever if I need it, set up my ssh keys, install git, swap ctrl/cmd, done. I can plug it into an external monitor and unplug it without is losing workspace/desktop context, I know I'm getting the best battery life, the resolution is working straight out of the box in all applications, no random slowdowns, etc. Meanwhile every time I try to put Linux of any flavor onto a macbook or a lenovo carbon or whatever, something gets fucky. Video drivers don't work, no webcam driver exists, the mouse glitches, the tilde key doesn't work, plugging into/out of an external monitor crashes it or causes all my windows to collapse into a single workspace, same issue for if the computer goes to sleep while connected to an external monitor, all sorts of weird things causes it to not sleep on laptop lid close and nukes my battery, battery life will drain excessively, arbitrarily, chrome will randomly cause the computer to slow WAY down, etc. Ok, that was a rant, but TLDR - I hate macs but coding on one just removes all the distractions from programming and let me "just do my job." (I am experimenting with windows as well right now because I love the Surface platform as a whiteboard and prototyper, so far WSL is great and I actually prefer it over linux because all the OS stuff "just works." Lots of weirdness around WSL still though)

tapatio 9y ago

This is most likely because engineers that use vi/emacs have to memorize language syntax and APIs as vi/emacs don't have code completion like most IDEs (at least not by default). And having that stuff memorized means you'll do better on a coding interview.

xena 9y ago
Fuzzy-matching autocomplete for vim: https://github.com/Valloric/YouCompleteMe https://github.com/Valloric/YouCompleteMe Fuzzy-matching autocomplete for emacs: https://company-mode.github.io https://company-mode.github.io If you are at all interested in trying something with all the bells and whistles turned on for you, check out http://spacemacs.org http://spacemacs.org
michaelmrose 9y ago
This is complete nonsense. It's completely trivial to get code completion. Its wholely unrealistic to suppose that someone would spend hours to days learning emacs but not spend 1 minute adding a completion plugin.
- sidlls 9y ago
  I don't use any completion plugin with my Emacs. Not coincidentally I'm generally a better reference for our programs' APIs and organization than the documentation or the other developers who use things like PyCharm and Sublime. For me not using completion works to keep me disciplined about the structure of code I work on. It also can make me vicious about arity and composability in code reviews. Fortunately most of those details are taken care of by design prior to implementation, so it's rarely an issue.
  - michaelmrose 9y ago
    I think not using code completion might have more to do with you having an excellent memory and it might not be fantastic advice for everyone but I would be interested if there were stats on usage of features like code completion in emacs.
  - 9y ago

tritium 9y ago

And yet, it's still all just plain text editing. Clearly, one kind of answer is more fashionable than the other.

look_lookatme 9y ago

I would not extrapolate too much from our broken system of technical interviews.

jawns 9y ago

Sample size is never directly mentioned, but: > At Triplebyte we evaluate hundreds of engineers every week. > I pulled the editor / language data for every Triplebyte interview over the last year. Based on that, if we suppose "hundreds" means "low hundreds," this suggests a sample size of 5K-15K. That's not a bad sample size for the total population. But if 35% use Python and only 5% use Ruby, then you're comparing a sample of 1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby data is more shaky than the Python data, and thus it's not a fair comparison. Same goes for a lot of the other metrics where the sample sizes vary. ~ Disclaimer: I'm no statistician (but I play one on Correlated.org).

tedsanders 9y ago
Be careful equating 'evaluate' with 'interview', when TripleByte has incentive to obfuscate. I suspect hundreds of engineers may apply through Triplebyte each week, but they may interview only a small fraction of those. If true, this makes your point about small sample size even more important. I was disappointed their charts didn't have confidence intervals or count numbers. Makes it harder to guess at how the numbers generalize.
jayant123 9y ago
Also, my guess is that the success rate for interviews would be fairly low, making any comparison of the success rates for different languages/editors even more statistically shaky. They should definitely publish the total sample size and success percentage for these results to be meaningful. Example: Say they hired 5 out of a 100 applicants who used Python, and 2 out of 20 applicants who used Ruby. That's a 5% success rate vs a 10% success rate, a 100% increase. I'd hardly call that a significant result though. Disclaimer: I'm no statistician either

minikites 9y ago

As so many Hacker News posters are fond of saying, correlation isn't sufficient to imply causation. Or perhaps their interview questions are designed in a way that biases a particular development environment (positively or negatively).

danjoc 9y ago
Or perhaps there's some racial bias and a certain ethnicity tends to be associated with Java and Eclipse.
praneshp 9y ago
> correlation isn't sufficient to imply causation. The article calls this out in the first paragraph.

CoreXtreme 9y ago

I've not read it! But experienced folks will be hired more often than inexperienced ones. And lots of old programmer use vim/emacs and other CLI editors. So, it's not surprisingly. Do they've data adjusted for programming experience?

hellofunk 9y ago

I started coding professional on Xcode. Then I tried a couple of other IDEs until my work in lisp required I give emacs a try. I eventually went back to other languages, mostly C++ and Python, but still use emacs. It really spoils you for text/code editing forever. Once you get passed the learning curve, every other effort to edit code or text in other programs seems ironically so archaic. What? I can't split the buffer quickly and get another one? What, I can't quickly bounce back to my previous editing window configuration, then bounce forward again? What, there is no visual undo tree? The list goes on and on. I now use the Vim emulation inside emacs which has been the killer feature in emacs to further boost productivity. And, last year I passed several rounds of technical interviews with one of the top 5 companies in SV, so perhaps I support the statistics presented here (though I did not get to use my chosen editor for the interviews, so they did not know I am an emacs user).

agumonkey 9y ago
I colleged on Eclipse. But I always had a thing for emacs (it has some parsing abilities oob, it's not just coloring and lexing). Anyway the day I finally clicked on the extensibility of emacs. Which was as trivial as defining a function, then defining a menu (which is duh.. a list). And have the GUI [1] updated I lost the remaining bit of love I had for Eclipse. I got 2 or 3 years of Eclipse JDT/PDT, all the talks about the power of plugins. It was nothing but pain. Then that old emacs gets me the same thing like it's breakfast. It hit me too hard. Lastly, I revived a win95 box, with Turbo Pascal. Another old and tiny blast from the past, and even though the 600KB IDE/compiler was amazing in most and so many regards. The fact that I was stuck with the default text editing features (no paragraph selection etc etc) made me feel what Emacs' freedom meant to me even on the tiniest instances. [1] note that I'm a keyboard fetishist, so I don't use 99% of Emacs GUI and knew most keybindings, I was just trying for the sake of learning elisp and emacs architecture.

todd8 9y ago

Fascinating, but what does it mean. I can dream up explainations, now after the fact, for this and the other interesting results in the article, but I wonder what the real reasons are? For example, Eclipse is the standard University Java course editor. Users that have switched to IntelliJ have probably had more time away from University and hence more experience. Is this the reason, or is it a "just so story"?

stupidcar 9y ago
Perhaps, but a lot of teaching is done with Python nowadays, and Python has a positive pass-rate, and actually tops the on-site success rate. So it doesn't just seem to be a case of beginners getting rejected a lot. Based on my anecdotal evidence of interviewing Java devs, I'd suggest the problem is the opposite. While there are some excellent Java coders out there, there are also an awful lot of "experienced" devs, with fifteen years of experience and salary expectations to match, whose technical ability proves to be pretty miserable.

deleted 9y ago

[deleted]

otakucode 9y ago

I think the headline kinda buries what I take to be the lesson from the data... companies don't like Java devs. Which is understandable, I imagine they find dealing with their ResumeAdapterFactoryInterfaceFactoryAdapterInterface needlessly gross.

optimusclimb 9y ago
That's such a dead and boring meme. Honestly the attacks I always see on Java (such as this one) always come off as completely naive. The amount of open source big data projects, and large profits and companies relying on Java technology should clue you in that this is unfounded. Hell, half of the modern smartphone market is all about Java. I don't think companies dislike Java devs. I'd also take an article from a company trying their hardest to commoditize engineers and turn them in to test scores with a grain of salt.

coldcode 9y ago

I have been coding for 36 years and currently do Swift/XCode. Used plain text editors back in 83/84 time frame but have using some kind of IDE since Turbo Pascal 1.0 and have zero desire to use either vim or emacs. I wouldn't hold a lot to the idea that old programmers use them more than young. People should use whatever they feel makes them more productive.

jhgb 9y ago
But (Gnu) Emacs is not a plain text editor. It can edit formatted texts, embed pictures and widgets etc. While XCode's editor, even if I haven't seen it in practice, most probably actually is. So it can't be the "plain text editor" aspect that is the differentiator in favor of XCode here.

weeksie 9y ago

If you're an emacs user there's nothing worse than doing an on-site interview where someone wants you to pair program using their editor. Nothing works right and you spend half the time flipper-handing it over the keys trying to remember how normal editors work. Then again, it's rare that I interview anymore. I've done maybe two of those kinds of (pair programming, someone else's rig) interviews in the last ten years and neither was for a job that I really wanted after spending time with the devs conducting the interviews. You want me to consult with you? Let me talk to your biz people to see if I'm the right person to solve your problems, and let the technical folk go over code I've written and we can deep dive on that if there is any question about my chops. Otherwise, fuck off. I'm too old to play "dance monkey".

psynapse 9y ago
"Let me talk to your biz people to see if I'm the right person to solve your problems, and let the technical folk go over code I've written and we can deep dive on that if there is any question about my chops." I feel you here. With becoming longer of tooth (in the trade) comes an understanding of how to contribute value via the multitude of avenues that are not just grinding out lines at the keyboard. I am surprised over and over at the incandescently bright young people who can wax lyrical about monad transformers, but to whom it does not occur to model a domain properly before cutting code, or who lack basic schema design and/or SQL knowledge. I am also in the boat of having vanishingly few technical interviews in recent years, but on a good deal of those occasions it has been obvious that the interviewer is just so not the person I should be talking to.
- quickthrower2 9y ago
  Modelling a domain is very underrated by companies as well as individuals and it's a shame because we end up with code bases full of mixed purposes language and confused naming that makes it harder to work on legacy. Monad Transformers FWIW are bread and butter in Haskell. So it a bit like going on about dependency injection. Not necessarily a sign of being stuck up or head in the clouds.
  - 9y ago

fcbrooklyn 9y ago

I'd be interested to know how their prescreening works. Sometimes those things are timed, and I can imagine a really short timeframe, say 30 minutes or so, might make the specific editor more relevant. For example, I absolutely hate Eclipse. Back when I was writing Java, I reached into my own pocket to buy IntelliJ because I was at least 50% more productive with it. I know there are people out there who dig Eclipse, but even if it's your editor of choice, it's more appropriate for dealing with a big, long-lived project. Similarly, if they had some existing project you are required to work with, full on IDEs like Eclipse might require a lot more fiddling to work properly, compared to straight up editors like vim/emacs/etc.

jorgemf 9y ago

Flame war ahead... I want you to bear in mind that this depends highly in the number of open positions per language/technology and the number of candidates per position. Does it mean I suck if I do Java in Eclipse compare to someone who does Ruby in Vim? Maybe they have 1 java position for 1000 java developers and 100 Ruby positions for 500 Ruby developers. So basically this article says nothing. It needs more information to be relevant.

bitwize 9y ago
That stuck out to me too. The "ideal coder" uses a Mac, Ruby, and vim -- pretty much a snapshot of the stereotypical Bay Area startup hipster. It could be that that's what they -- or their clients -- are selecting for.
compumike 9y ago
(Article author here -- I'm one of the interviewers here as well.) Our technical interview is really designed to look for your strengths, such as being productive while writing clean, correct code in any language/environment of your choice. If you do well at that (or excel in other sections of the interview such as system design or knowledge questions), we're going to work with you as a candidate, regardless of the language you choose to interview in.
- jorgemf 9y ago
  Still is irrelevant. If you have 1 java position and 10000 Ruby positions, you are going to interview more Ruby developers, your data is going to match more closely the reality. For the 1 java position you are going to be more picky and discard more candidates. But even with the same number of candidates and positions your test are going to have bias, how do you know the java test is as difficult as the ruby test, it could be more or less difficult. You can do a proper analysis fixing some of the variables. For example you can fix the language and compare how the different editors affect the selection process. But still you have to show other variables as the years of experience in the language. And finally you have outlayers like who did an interview with you but was discarded because of living in a different country. But I guess I might have enter into the statistics.

pfarnsworth 9y ago

I find it kind of funny and rather llazy that they don't dig into deeper as to why there's such disparity between programming languages. To me, it's probably that the questions are geared towards Python/Ruby, rather than C++. For example, parsing a CSV is one line of code in Python/Ruby by default, but if you're supposed to do this in C++, that alone can take 10 mins to write and test, etc. If they really want to make their test/evaluation fair, and if they truly want to be inclusive in the languages, they should figure out why there is such disparity, it's probably with the questions they ask rather than the candidates. To me, looking at these numbers, it just means that were I ever to interview with them (I never will because I think they offer no benefits whatsoever, since I don't want to interview with them, and then with the external companies again when I can just go with the external companies the first time), I would just choose Python or Ruby to interview in, since the tests appear more geared to those languages.

WhitneyLand 9y ago

Programmer interviews are broken. Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code. Technical hiring processes harm both excellent candidates who don't interview well and companies trying to hire good programmers. Agreed. Great start. [make an] investment by going through our interview process, afterwards we can fast track you through to final interviews...O(n) process into an O(1) process Makes sense. Use of complexity notation is transparently gratuitous, but I liked it, because it shows the intention to understand and connect. Traveling the world most people appreciate an honest attempt to speak to them in their language even if you're not fluent. You'll have to take a coding quiz with questions in several different languages, as well as some design questions. The coding questions involve reading blocks of code, and answering questions about them. Intuitively I hate the idea because I'm skeptical of how effective it is. Do you have non-anecdotal data to show how well your quiz correlates to specific goals? After the quiz, you may be asked to complete a coding challenge... What triggers that requirement? Also same question as previous question about correlation/validity. Then, you will move on to our technical interview... What do the tech interviews consist of? They can vary wildly across companies, interviewers, and formats. ...work history [is] meaningful but relying solely on [it] results in missing good programmers This is a show stopper for me. If a company doesn't acknowledge that what people have accomplished with previous contributions, projects, development work is not just meaningful, but extremely important and a top criterion to find great people, then I don't believe they have the correct perspective. But hey, I can be wrong about things, I'll try to be open minded. Point me to solid data showing tests and quizzes are a better predictor than accomplishments and I'll have a look.

mal808 9y ago
Regarding the work history thing, I think the point is missing good programmers who have a patchy work history. I, for example am 42, but I've only been a programmer for 3 years. Someone looking purely at my work history probably wouldn't have bothered to interview me for the first couple of years. Fortunately for me that wasn't the case.
- WhitneyLand 9y ago

kybernetikos 9y ago

When it comes to a choice of programming language, ultimately a right tool for the right job mentality is good, and for the kinds of tasks that commonly come up in interview situations, C++ (or even Java) is not usually the right tool. This of course doesn't mean that for real life situations, they're never the right tool, but if someone chooses C++ to solve a simple tree inversion or sliding window problem in an interview situation it probably means that they don't have sufficient familiarity with any language more appropriate to the situation.

dabei 9y ago
C++ in the right hand is very succinct and expressive for solving algorithm problems. If someone claim c++ expertise but switch to their secondary language for some questions I'd think twice about hiring them.
- kybernetikos 9y ago
  This is true - there are some people who can make C++ sing even for these kinds of problems, however most people who attempt to use C++ in an interview situation to solve toy interview problems do poorly. For most people, it takes them longer to solve the problem, to find and correct bugs and to provide tests that prove that they've solved the problem correctly than someone solving the same problem in a different language. I've noticed this in my own interviews, and this article mentions it too. It seems to me that the most natural explanations for this are either that the pool of C++ programmers has more bad programmers in it than other languages (which I don't believe) or the language and tools themselves are not as well suited to solving interview questions in interview situations. As to penalising people who switch languages for problems, if you give guidance that you want to see a solution in a particular language, that's fair enough, in any other case it seems unfair and misguided. Perhaps you don't really agree with 'right tool for the right job' when it comes to languages?
drharby 9y ago
But google told me that my language choice won't impact their hiring decision :(

c3534l 9y ago

The low score of C++ and high score of Python in pass rate made me think for a moment. How are they determining pass or fail rate? Because if they're failing people because they forgot a semicolon or something trivial like that, then this ceases to be a test of interview performance and more a test of how many inconsequential mistakes does the average person make. So for instance, Vim doesn't provide much help when you mess up. If you type a variable name that doesn't exist, that's your fault (I'm sure with plugins you can get more). So do Vim users do better because they're nerdier, don't mind a steep learning curve, and like working in a terminal, or is it because Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"?

xiphias 9y ago
In my experience the new ,,cool'' startups prepare interview questions in python/ruby/..., give some time, and don't measure the time spent by the program. In this setup writing the same code in C++ (including the right STL headers) is a pain in the a. I remember that when I selected C++ as my language for interviewing, I was strongly punished, because my interviewer didn't understand why it takes 10 minutes to do something relatively simple (that is just a few line of code in a higher level language). After that I switched to using Ruby in my interviews, and passed easily.
rch 9y ago
Their entire sample seems to be composed of people who are willing to click on a green button and "take the quiz" as part of a job search.
thomastjeffery 9y ago
> If you type a variable name that doesn't exist [in vim] Start typing a work and press C-x C-[nolk] * n for a word in open buffers * o for "omnicomplete" which is quite similar to "intellisense" * l for the entire line in open buffers * k for the word from a dictionary There are several others, but those are the main ones. As soon as you start a completion, you can use C-p and C-n to go up and down the list respectively. > Vim doesn't provide much help when you mess up Vim is designed for editing text. I don't know if a better editor (yet) suited to editing, rather than writing. > Vim doesn't tell you that you accidentally called "reduc" instead of "reduce"? It can, so long as you have it set up to. Vim even has a spell checker.

didibus 9y ago

This is missing data about the job role. Are python devs hired as web dev, and c++ as 3D engine programmers? Does the triplebyte hiring process not take into account experience and role? Don't they expect more for certain roles and a given experience?

dllu 9y ago
It seems Triplebyte just tests everyone on web technologies regardless of role or experience. It makes sense that C++ engineers are less familiar with large scale web apps, but are valuable hires in the industry for robotics, finance, game engines, etc.
- compumike 9y ago
  (Article author here.) I'm one of the interviewers. If I had to guess, maybe a third of our generalist-track interview is web-related. I've personally interviewed many engineers who did not know the first thing about any web stuff, but were particularly strong on other sections, so we accepted them. You really don't have to be great at every section to pass the Triplebyte interview; you just have to show your personal areas of strength. And we give you lots of different opportunities to do that. Of course, we will then match you with appropriate roles and avoid the ones that require web-specific skills. FYI, we already have robotics, financial, and game companies on our platform!

Rantenki 9y ago

This whole article reads like an celebration of confirmation bias, which could be summarized as: "The people that pass our interviews tend to be the ones that use the tools that we prefer." I don't think there is any value in looking at this blog post and making any conclusions about which editor and/or scripting language to learn next. You might go interview at a .net shop and get the opposite result from the one that you wanted. Disclaimer: I personally match their preferred technologies very closely, but work with some really smart people that write C#/F# in VSCode.

davesque 9y ago
These were my thoughts exactly. Other things can also come into play. For example, what if certain technologies demand higher levels of skill? Then the interview pass rate might naturally be lower because the bar is higher. There could be countless ways that these numbers actually mean something different than what is implied.
cableshaft 9y ago
There are a lot of really smart C# programmers out there... but there are a lot of subpar C# developers out there too, especially outside of Silicon Valley. I've had a boss who brought in at least 60 people in (while I was there) for an unsupervised (unless you had a question) 60-90 minute test, where he'd leave them alone, let them google, etc, but would sit down and discuss what they did and why afterwards, and the amount of people who couldn't get past the first FizzBuzz question was pretty high. Those that got past that, the number who didn't know the basics, like what an Interface was or how they worked (or how you're not supposed to write implementation code directly into them) was higher still. And these are people who say they have 5-15 years of direct experience on their resumes. I'm not a huge fan of tests and I do tend to struggle myself, but not that badly, and not when unsupervised.
- kevindqc 9y ago
  I never understood and that can happen. What have they been doing for 5-15 years? I can understand performance anxiety being a factor.. but 60-90 minutes and still can't answer FizzBuzz?

wazoox 9y ago

In my experience with many interns in the past few years, most of them came using Sublime Text, in the unpaid nagware version; however all of the most promising and intelligent ones moved forward to other editors, most of them going to Vim.

mulmen 9y ago

This screams of lying with statistics. There are no interesting conclusions drawn here, just some correlation that is at risk of being based on selection bias. This is just fodder for the endless tech variants of ford-vs-chevy debates. The elephant in the room is that we don't even know if technical interview performance is a good selector. I'd be a lot more interested if this data included long-term results of actual job performance like employee and manager satisfaction, time in role, time to promotion, etc.

seizethecheese 9y ago

This just makes me wonder what factors are correlated with Eclipse editor usage. Perhaps there's a sub-optimal channel to the developer career path where people are taught Java on Eclipse. Does anyone know?

walshemj 9y ago

the OS break down was interesting as I would bet that most of the jobs are using Linux's to run the code on the massive over representation of OSX is odd. Buying a more expensive Mac to use just as a terminal just seams an odd choice.

mizzao 9y ago

Aside from the correlation !=> causation issues, there's a ton of endogeneity and selection bias in this data. The initial programming quiz has code snippets only in Javascript, Ruby, and C. The interviewers are usually more familiar with more modern technologies. Indeed "enterprise" (C# or Java) developers tend to do poorly at TripleByte, and Java is highly correlated with Eclipse usage. TripleByte also recruits primarily for startups which tend to use certain technologies. I'd say the final stage offer data is so biased by the selection process along the way that it's almost impossible to draw any conclusions from it.

arjie 9y ago

Interesting. Triplebyte is successful if the candidates they recommend are candidates the companies hire. That means that by the time a company sees a Triplebyte candidate, the candidate's language/editor choice should have no signal left in it if Triplebyte has perfect performance. Quite cool that they're willing to share fairly transparent performance metrics with us.

Karrot_Kream 9y ago

I'd love to know whether these percentages were normalized by the number of people in each bucket. i.e. we should calculate P(pass | <emacs>) or P(pass | <c#>) and see how that compares to P(pass) as a whole.

compumike 9y ago
Yes, that's already what we're showing. On the "Triplebyte Technical Interview Pass Rate by Editor" chart, Vim is shown as +38%. This means that: P(pass | vim) / P(pass) = 1.38
- deleted 9y ago
  [deleted]

btilly 9y ago

I would love to see error bars on all of the graphs. Since we're estimating a pass/fail ratio, any of the confidence intervals in https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval https://en.wikipedia.org/wiki/Binomial_proportion_confidence... could be used. (Jeffreys interval may be a good default.)

dsacco 9y ago

With respect to languages: is it the case that Ruby programmers do better, or are companies hiring Ruby programmers more lenient? (Ruby being an example; Python also applies here, whereas the converse would be a hypothesis that C++ programmers are not less skilled but rather are judged more harshly in interviews).

luord 9y ago

> if you walk into any tech company in San Francisco you’re inevitably going to count far more than 60% MacBooks This surprised me. It might be because I'm not from the USA but in every company I've worked for on-site (including the current one), the overwhelming majority of developers used and preferred linux. That aside, I'm glad python does well.

caconym_ 9y ago

Interviewing in Python for my semi-dream job is one of the best decisions I ever made (on par with switching from Emacs to Vim). I don't even know the language that well, I'm an intermediate user at best, but the the reduction in cognitive load from how straightforward and ergonomic it is is crazy. It was also nice that they had an editor available with serviceable Vim bindings. I think it was a big improvement over coding on a whiteboard.

KirinDave 9y ago

And here I thought we had hit peak senseless-pattern-matching with the tabs-vs-spaces dataset. I was hoping for a quick section introspecting on correlation vs causation and what this says about their process as opposed to a semi-serious presentation of predictive power. I can only assume a total lack of rigor and process if this is what they choose to talk about publicly.

tarboreus 9y ago

The post is an enjoyable one, but makes a couple of assumptions, one being that experience and coding ability are what get you hired. Sure, they help, and they're a prerequisite, but showing up wearing the right clothes and having the right attitude might actually matter more. People who use OSX are more likely to be the kind of dev with the tidy beard and the square glasses and the decent shirt. Devs using Emacs in Linux are more likely to be the aneruotypical who doesn't look their interviewer in the eye. Both are more likely to be good coders, though, since you don't get taught Vim and Emacs in school, you have to pick it up through a non-school network or through self study and research. Eclipse is the kind of thing you use because your professor taught you Java and you never did any coding that wasn't for class. Sublime is the kind of thing you use if you just don't really care about your editor one way or the other. I'd be curious to know what kind of shop they are at TripleByte. Something tells me they might be Ruby coders using OSX all the time. Like attracts like, especially in interviews.

deleted 9y ago

[deleted]

linguistbreaker 9y ago

This "data" is not very meaningful. It says much more about Triplebyte than about developers, technologies or interviews.

brisance 9y ago

Seems like machine learning could be used to provide greater insight.

AzzieElbab 9y ago

Someone ought to put together performance charts by Zodiac signs