11 ms·
The Silent Majority of Experts
- smeg 14y agoPeople still building software in Perl ought to be arrested.
- waterlesscloud 14y agoI know a guy working at JPL doing Perl stuff for internal use still. I give him a hard time about it, but it's a lot cooler than Social_Widget_0003993986 in any other language you care to name.
- smeg 14y ago"but it's a lot cooler" Are you referring to the work or Perl? Most programmers I know working in a scientific area switched to Python years ago and run away screaming when they hear the work "Perl".
- inopinatus 14y agoChoice of language is quite marginal when it comes to determining the quality and utility of code.
- lubutu 14y agoI'm not sure that's true. Though it obviously depends more on the programmer themself, the semantics of a language can make it easier, at least, to write higher quality code. For example, bounds checking removes a whole class of bugs (probably the most common in C) from the equation. I disagree that this effect could be described as "marginal".
- batgaijin 14y agoyeah but other languages have better minima cause they hand-hold so much, like python
- stcredzero 14y agoActually, it's been known for quite awhile that there's a factor of 2 to 3 times in productivity. That's not really marginal. Talent far outstrips that, though.
- pessimizer 14y agoproductivity != quality and utility
- stcredzero 14y agoIt doesn't equal utility, but it does equal quality, at least to the extent that bugs per SLOC matter. This has also been known for awhile, with data to back it up.
- inopinatus 14y agoIndeed talent does. I make no claims about productivity. Your note is intriguing, references invited.
- zaphar 14y agoI cut my teeth on perl and don't run screaming when I see to this day. Perl has an undeserved bad rap. I like python as much as the next guy but frankly Perl does some things way better than python. And don't get me started on Pythons rules around scope.
- zerostar07 14y agoBlogging also takes confidence, that you have something substantial to say and that what you say is valid. Unfortunately, due to the Dunning-Kruger effect[1], actual smart people have too much self-doubt, which would imply that not-so-smart people are overrepresented in the blogosphere/google search/messageboards. Couple that with the internet's popularity-contest dynamics and you have a larger problem than earlier publishing media (which required a substantial level of peer-review-by-experts)/ Take home message: read everything on the net with a grain of salt. [1] http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
- delinka 14y agoAlso, writing requires a tremendous amount of effort. Effort I'd personally rather spend on creating my next project. When I get to a point that I feel writing would scratch an itch, I find that simply getting the thoughts out of my head and onto paper or into the computer is enough therapy for me to move on. Organizing that into coherent written English takes lots of effort I'd rather spend elsewhere; sharing it on the 'tubes then necessitates dealing with the things you mention. In short, being technical (and pretty decent at it at least), public writing just Isn't My Thing. That said, I find that spending a few moments organizing my thoughts into coherent English for commenting on HN encourages me in the direction of writing and could indeed possibly lead to be blogging. Eventually.
- NickPollard 14y agoI enjoy writing when I get the chance, but right now, working on my own software project, it's hard to take time away from developing in order to write blog posts. This even though I know that raising awareness about what I'm doing is as important as actually building the product. I look with awe upon people who manage to create amazing things whilst also writing eloquently about them - I just can't handle both at once.
- carlio 14y agoExactly. Writing well is hard, and you have to spend a lot of time thinking, writing, editing. 'A writer is a person for whom writing is more difficult than it is for other people.' - Thomas Mann
- yen223 14y ago"An appealing theory that gets frantically upvoted may have well-understood but non-obvious drawbacks." I'm reminded of the quote, "To every problem there is a solution that is simple, elegant, and wrong." I have fallen into the trap of looking at a long, messy piece of code, thinking "I can do better than this". I would replace a 50-line code with 5 lines, only to have it fail at some random edge case, which the original has been fixed for. That is why I always remain skeptical of people who are out to "disrupt" an industry, especially when they don't have much experience in that industry.
- jeffjose 14y agoCant remember where I read this, but I realized a while back the folly of "rewriting your code". Your code looks messy because it works. Its the result of weeks of testing and bug fixes. It can handle special cases and is robust. Many a times, to start of, a clever 5 or 10 lines would suffice. But the moment you start testing it for production, it'd require more and more logic to handle _all_ the cases.
- TimJRobinson 14y agoI think this is very common with new simple magical frameworks / languages too. They work really well for most common tasks but can't handle the edge cases.
- samnardoni 14y agoI'm pretty sure you're referring to a Joel Spolsky article, "Things you should never do". http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html
- jawr 14y agoExcellent post, thanks for the link.
- deleted 14y ago[deleted]
- mjn 14y agoIncentives also tend to disfavor "popular" work like blogging, but I think in parts of academia that is somewhat changing. May be changing less slowly with regard to experts in industry, some of whom also have confidentiality issues to worry about. As one example, law-professor blogs have really taken off. They're good for building reputation and making a legal argument actually have real-world impact (people read them, whereas the average law review article is little read), and sometimes have even been cited in court decisions! So there is a movement towards law professors doing more popularly oriented, but still legally sound, writing; and towards universities taking this into account. Science has also long had some research/popular-writing crossover, and there are a number of researchers who blog in areas like physics and climate science. And mathematics has an increasingly strong blogging culture, with Terence Tao setting the nearly impossible to top example. So I think there is good stuff out there if you know where to look. Of course, it's also good to experiment on your own, and also to look elsewhere (e.g., in books).
- dkhenry 14y agoIs it a silent Majority? I don't think anyone will doubt that not every interesting thing that gets done gets published and discussed on the internet and on blogs, but I don't know if I would say its a majority. From my experience there are some people in the software world who think they are either not noteworthy enough to write about what their doing or they are too noteworthy ( I am not going to share my amazing super advantage I got by making this really smart and novel thing). I have found however that in general software development, like any technical trade, works best when its subject to peer review and an open community. I think most people also know this, which is why the tech heavyweights have talked about their infrastructure and technique. They do it so others can build on what they have done and as a community we can build on each other. There are a few who don't care to participate thinking they are either above it or below it , but the truth is most people do and everyone should.
- pmjordan 14y agoI've worked with far more excellent people who don't blog than with those who do. In many cases, confidentiality agreements prevent the really cool stuff those people work on from seeing the light of day (except of course nestled deeply within the binary of the product in question, if it's even for public release). I don't have data, but anecdotally I'm completely willing to believe it's the majority.
- dkhenry 14y agoThat really cool stuff nestled deep in a propiratary binary, Odds are its not really that neat or cool. Thats not to say the people who made it aren't really smart and good at what they do, but odds are its been done before. By refusing to learn from and give back to the development community all they have done is made sure no one ever learns from something they might have done differently or better. Meanwhile the experts in the field are the ones who have taken whats been done, understood it, and built upon it. My point is a majority of those people _are_ giving back in the form of public discourse.
- eliasmacpherson 14y ago
- alan_cx 14y agoThere is a more general point here. There seems to be a growing assumption that activity on the internet is some how proportional to activity in real life. Yes this could be a problem for programmers, but IMHO, its becoming more and more of a problem with news gathering. Issues or causes with facebook and twitter accounts get far more coverage than those with out. Basically these sites provide tittle tattle to fill the 24hr news channels. OK, cool if you happen to be in a banana republic dictatorship with internet access, but deep in Africa, your tribe can be slaughtered and no one knows.
- 16s 14y agoI liked this sentence. It's so very true. "Just because looking down your nose at C++ or Perl is the popular opinion doesn't mean that those languages aren't being used by very smart folks to build amazing, finely crafted software."
- pjmlp 14y agoVery true. While bringing lots of money home, I would add.
- konstruktor 14y agoI remember, but sadly haven't bookmarked, a blog post by a guy who made a fair share of money building a decent music player for BlackBerry when all cool kids were building iPhone apps. Actual needs that will get people to pay money can be quite unhip.
- konstruktor 14y agoAnd you get a lot of attention for bashing something, which aggravates the problem.
- pjscott 14y agoWhat problem? You can bash a language without saying anything negative about its users. Not all programming languages are equally good, for any reasonable definition of good. This is a valid thing to discuss.
- rbanffy 14y agoThe same can be said about COBOL. Or Visual Basic.
- SilasX 14y agoOr PHP (shudders), which Facebook (for some reason) relies on.
- shrughes 14y agoImplicitly in contrast is a loud minority of newbs, with their resume-building blogs that have an about-me page, a picture of themselves, and reposting/paraphrases of old Joel on Software posts.
- tomrod 14y agoHey, we newbs can contribute SOMETHING--namely useful content for other newbs (that rises above the noise). Newbs have to learn somewhere, and if experts aren't contributing, then to whom do we turn?
- tomrod 14y agoI'm surprised to see the above comment got downvoted. Do people think that excited acolytes contribute nothing?
- zumda 14y agoI agree with you. There are things that the experts don't want to talk about (because it's too easy) and then there are things that a beginner can explain in a simpler way than an expert. Staying in a beginners mind is hard once you have spent some time with a technology. It also sometimes offers great points for the experts what could change, because it could be easier or is simply unclear. Of course there are very simple-minded blogs with practically no content, but there are also simple-minded Github accounts that offer practically nothing. There are CVs for the same purpose. I don't think this has anything to do with blogging per se.
- FrojoS 14y agoAs one among them, I agree. As far as I know, most Wikipedia articles were mainly written by students, not experts.
- lani 14y agoif only I'd read this article two - maybe five years ago ...
- rbanffy 14y agoI signed so many NDAs in the past 4 years it almost looks like I did nothing.
- OmegaHN 14y agoI think this problem is why academia is so valuable; academics sole purpose is to share their discoveries. What they build is not a product, but rather a set of ideas ( or others' ideas) to share with the public
- dsirijus 14y agoOne thing is being a good programmer and wholly another selling oneself as good programmer. And that's absolutely not sad state of affairs; it is how it is and it is how it should be.
- Wilya 14y agoWhy is it how it should be ? The fact that I know how to sell myself makes me better paid, better treated, and better recognized that others who have the same level of programming ability as me but are less good at shouting it to whoever wants to hear it. And yet, I don't provide anything better than them to anyone. Worse, it's overall detrimental, because I spend time making myself known instead of doing actual stuff.
- dsirijus 14y agoWorking with/for someone isn't programming, it's business relations, and thinking otherwise hurts both parties. It is a nuanced difference, though.
- Wilya 14y agoErr, yeah. Except at the end of the day, the goal and outcome of the relationship is the programming (or, well, the technical side in general). The business relationship is supposed to be just a construction between us, in order to be able to agree on the tech work. If there is no need on the other party, they have no reason to do business with me. People don't just hire me because I'm a cool guy and I say funny jokes and blabber insightful-sounding things. They hire me because they believe I can, one way or another, provide them some value. Telling jokes, or writing link-bait-ish blog posts, (or doing mostly shiny and maybe useful software), or whatever you want to put under "developing your personal brand 101" is just a way to make them believe that in order to put myself in a position where they would allow me to provide value to them. It doesn't, in itself, provide value to anyone. Or, at least, it provides less value than the time I spend doing the real, and boring, and lonely, and hard to justify research groundwork that could (or not) lead me to developing something valuable. And I mean research in the broad sense, be it evaluating new or old software, trying to talk to people and to discern if they have problems that I could solve, or just chasing after a sneaky bug in an obscure backend thing that I happen to maintain.
- gmurphy 14y agoWe call this the 'dark matter' problem - when you're trying to hire great people, for every well-known rockstar, there are tens of people who are better, toiling away in obscurity. The early JavaScript scene was like this - there were people who published amazing JS work and everyone on DHTMLCentral knew who they were, and it was easy to find them and try to hire them. Yet we felt this huge frustration because occasionally we'd interview someone who was as good or better, but we had only found them by accident. I'm excited about sites like Dribbble and GitHub, which make it easy to showcase your work without having to go through the pantomime of a large folio piece. Though it can be a pain for introverts to do, I've never met anyone who regretted overpublishing their stuff.
- sanderjd 14y agoI think that GitHub merely shifts the "dark matter" problem. I think your assertion about "rockstars" applies to GitHub just as well - for every person with an impressive GitHub account, there are tens of people who are better, toiling in obscurity. The point is that you can't just look for people who are famous or prolific on blogs or twitter or GitHub or whatever else.
- njharman 14y agoExactly. Github (and every other flavor of the time) is an example of the problem. Not the solution.
- vm 14y agoNah, Github is a step in the right direction. They make publishing online easier which gets more people to do it. It's like how twitter grew the blogging world by making it easy to do. Sure, not everyone does it yet, but it's better than before
- tomjen3 14y agoAlso with Github you are publishing actual code, and not just writing about code (I am looking at you, coding horror).
- tubbo 14y agoI think this has more to do with age than anything else. But it might also be because you can do stuff faster in Ruby or Python or JavaScript so you have all this time to wax poetic about it. ;) There's still nothing like C/C++ for getting close to the metal. And until we have an OS written in Ruby or Python, rather than C, it's gonna be like that forever.
- tikhonj 14y agoIt's funny you should mention that, because the language in the blog post was Forth. And you can run Forth directly on a chip, unlike C. I'm pretty sure that's exactly what GreenArrays is doing[1]. Starting this fall, I think I'll be working on something similar as well. [1]: http://www.greenarraychips.com/ http://www.greenarraychips.com/ Or what about lisp? While these were well before my time, my understanding is that they could run Lisp directly in hardware. And, of course, there have been OSes mostly written in higher level languages than C or C++. The one that springs to mind immediately is the Singularity OS[2] project from Microsoft that is mostly implemented in C#. I think there is still some assembly, C and C++, but the kernel and drivers are in C#. [2]: http://en.wikipedia.org/wiki/Singularity_(operating_system) http://en.wikipedia.org/wiki/Singularity_(operating_system)
- n0on3 14y agoThe thing is: (blogging|writing)-about-/publishing- code takes time. And time is really what people doing amazing stuff are short on the most. Don't get me wrong, both discussing and publishing are amazing, but often life is just too short for the blah blah blah, unless you have good reasons for doing that.
- nathancahill 14y agoNo doubt a lot of expertise is locked away in commercial software, but the open source community has equally (or more) impressive experts working on software. Maybe not blogging about it, but the expertise is definitely there.
- dkhenry 14y agoIf the point of this article is to say that you should not be spending time worrying about and evangelizing your tools and more time using them to create awesome stuff then I agree. However if its to say that there is a large crowd of exceptional programmers and engineers that no one hears about because they spend _too_ much time creating great projects then this is baloney. There are some people out there who are truly exceptional and might never be heard from. However in general the real exceptional people are the ones pushing the envelope of whats possible forward. Their work _always_ finds the limelight. The author uses an example of a rigid body physics implementation in a game, and while that's great work and the team should congratulate themselves its not that big of a deal BLAS had been around for decades by that point. That does not describe experts doing amazing things. It describes software developers doing a good job and not being cocky about it. I think why this has received such a warm welcome here is so many of us software developers want to think we are part of that silent majority. That we are doing amazing things too and if we cared to we could write articles and become famous. The truth is we are not experts we are people who have jobs to feed our families and we may be just as skilled and talanted as the vocal minority, but we would never be able to reproduce their success. That's not a bad thing. I have no problem being an unknown developer working on a piece of corporate software and providing a good living for my family. Its ok if others get press time for their contributions I might take from them and learn, but it doesn't change the fact that they at the end of the day have moved our profession further and I haven't.
- sshumaker 14y agoYou're wrong. There's only a mild correlation between fame and skill set. I can name a few dozen 'famous' internet developers, and have even interviewed a few of them, and most weren't anything special, other than being good bloggers. Meanwhile, take someone like Jeff Dean (who I just learned about a few days ago). Designed and implemented BigTable, MapReduce, Protocol Buffers, etc. I'm not sure most people outside of Google have even heard of him. Sometimes your work finds the limelight but you aren't credited for it. We all know Elon Musk but who knows the guy who actually wrote the code for the rockets? I don't remember the name of the Amazon Dynamo guys, and have no idea who designed EC2 and S3. These are awesome systems used by millions of people, but the creators are (relatively) unknown.
- kamaal 14y agoEven as I'm writing this I'm at a hacknight in Bangalore, a over night even on BigData. We are hacking away to learn and make a lot of things. One thing that I observed today was an old friend and colleague sitting next to me. He sort of wrote a Perl program to analyze a weeks worth of tweets in India and draw real cool visualization out of it. The thing is script was not really big and was full of functional programming stuff. As I was reading the script with another friend, My friend was like 'Oh! I though SQL is unpopular these days, or Perl was dead, or no one uses functional programming these days'. I soon realized- For each of these cries, some one is still sitting and doing awesome stuff with things and using the most simplest and basic tools. Vast majority of awesome guys don't blog or tweet or harp about every bit of tiny things they have achieved on the Cyber space. But your everyday guy is vastly influenced by ranting, blogging and tweeting and takes them to be holy! And anything other thing than that- which produces results is generally a 'black swan' event to him. So when something unpopular is used to get the job done, its often difficult to the everyday guy to understand this.
- SoftwareMaven 14y agoMy experience is that less experienced programmers tend to want to get things done the "right" way, so they tend to jump on the latest-and-greatest bandwagon. After enough experience (and having seen multiple bandwagons come and go, they tend to just want to get stuff done. This change in focus is one of the most valuable things (IMO, of course) that more experienced developers bring to a project.
- pertex 14y agoI've been fascinated by this phenomenon for a long time. The best coders I've found are not the loudest voices. In many cases they do not comment publicly on the web at all. They just submit bug fixes. That said, the best code I've found is the stuff that's made publicly available via internet, and I'd argue it's precisely because it allows all those "silent experts" to review it and submit their fixes. It's not the best simply because of the contributions of the silent experts, but because their scrutiny raises the bar. It takes a certain amount of confidence to make code avaiable for all to see. Because it becomes very clear, quite quickly, whether it's any good or not. I don't see this same type of peer review and vetting within the corporation. The goal there is different: one need only satisfy the company's clients, most of whom cannot themselves review code quality. In my experience, many of the silent experts who submit fixes work at corporations. They are no strangers to closed source. So it's all quite interesting. Open source and open discussion in its own strange way effectively raises the bar. But at the same time the web also lowers the bar to spreading bad code. And disseminating much nonsense chatter about programming. Hence we have lots of chaff. Lots and lots of chaff. The wheat is there, but you have to know where to look. And generally, when you find wheat, you'll also find lots of "silent experts" submitting fixes. I often wish which they were the ones making the noise on the web. Because then it wouldn't be noise. Many of these guys really know their stuff. I guess over time the web has worsened things. In the early days, many online contributors, e.g., to bulletin boards and newsgroups, really were knowledgeable in their chosen fields. There were just less people online. That seems to have changed somewhat, but maybe it's just my perception. In any event, they might be silent, but we know they are paying attention. No one is going to completely ignore the continuing growth of open source and open collaboration.
- TwoBit 14y agoIMO this is why people who post online think GCC is more popular than it is. VC++ and C# have far more usage than GCC, but most of it is by professionals who don't post online. Ditto for the lack of awareness of the popularity of Perforce in professional environments.