7 ms·
Why isn’t there a glut of good software engineers?
- coldcode 14y agoWhy isn't there a glut of baseball players who hit .408 in the major leagues? Because it's a rare person who can hit a ball thrown at nearly 100 MPH 40% of the time. Software engineers are common but people who deliver quality product after product is a rare bird. Programming at a high level is not as easy as this author seems to think.
- pekk 14y agoEvery company seems to think it needs programmers who hit .408 in the major leagues. Maybe the problem is that they think making money is a zero-sum game like baseball, so that they have to have the "best team" and not just a team which can build what's necessary to make some money (not the most money, that would also be a game).
- thedufer 14y agoYou're assuming that 2 people hitting .200 are equivalent to one hitting .400. This isn't true in baseball, nor in software development. The competition is because the person hitting .400 is worth at least an order of magnitude more than a person hitting .200. The same discrepancy is not found in salaries, so you obviously prefer to hire from the top.
- supercanuck 14y agoThe average fastball in the majors is more akin to 88-90mph and fastballs make up about 70% of the pitches thrown. Accuracy is a better indicator of success for a baseball pitcher than velocity.
- jpatokal 14y agoPhilip Greenspun, aka "this person", has a pretty solid resume when it comes to programming. http://philip.greenspun.com/personal/resume http://philip.greenspun.com/personal/resume
- npsimons 14y agoGreat coder can't understand why there aren't more great coders around. Film at 11. Perhaps Greenspun has "suffered" from working with too many great coders, or finds programming easy, so he can't understand why there aren't more great coders around. Maybe if he studied more sociology, or IQ statistics, or hell even economics (in the form of market demands) he'd be better equipped to understand why there aren't more great coders. I know who Greenspun is, and I have a lot of respect for his programming chops, but this comes off as someone either disconnected from reality, or someone thinking that because they are expert at one thing, they are expert at everything.
- djKianoosh 14y agojust because "this person" might be solid still doesn't mean anything. quite possibly it might actually delude him from reality if he's way above average
- sixbrx 14y agoAnd like many brilliant and talented people, they're completely clueless that in the real world, many other people just don't have the natural abilities that they themselves have, thus they are repeatedly perplexed that others just can't do the "easy" things they do. Often I've seen instead that they will instead ascribe others' failures as moral failures, which bites doubly against those that are trying hard but can't measure up.
- beat 14y agoI struggle with that myself. My attitude toward people who aren't accomplishing much with their lives is "Well, work harder!" But while much of my own success is due to hard, focused work, much of it is also due to raw intelligence and talent. Not everyone has that.
- danielweber 14y agoHe also wanted to make his employees work 70+ hour weeks: Suppose that a programmer needs to spend 25 hours per week keeping current with new technology, getting coordinated with other programmers, contributing to documentation and thought leadership pieces, and comprehending the structures of the systems being extended. Under this assumption, a programmer who works 55 hours per week will produce twice as much code as one who works 40 hours per week http://philip.greenspun.com/ancient-history/managing-software-engineers http://philip.greenspun.com/ancient-history/managing-softwar...
- bigdubs 14y agoThere is another force in the market that the author is missing; there are a lot of quality engineers doing their own thing or starting their own company.
- jbcurtin2 14y agoIt just irks me when people come to me and offer me an idea, then say they'll give me 1% of a company that I build all the software to it. Either I don't understand why that happens or it's just plan greedy. I'd rather work in a company that pays me well now while playing with my own ideas in parallel.
- solistice 14y agoAnd likely, they'll want you to take over the financing and distribution, as well as the marketing and accounting as well, because those are all hard things they don't want to deal with. They just want to have the idea. And they stay blissfully unaware that with just the idea (and maybe a patent pending), it's be 3-10%....of wholesale price for them.
- ben336 14y agoThe article starts with the primary assumption that programming is a commodity skill, then is astounded when the world doesn't back up his theory. Not exactly going to learn a lot like that.
- tempi35 14y agoWhy programming isn't a commodity skill? Or will become such commodity soon? Programming is like a house builder. Yes there are better and more experienced builders, but most of them can build a house top to bottom with no problem.
- paulbaumgart 14y agoProgramming is not like building houses. The computers on which the software runs are the "house builders"-- the parts of the system doing the repetitive work. Programmers are the architects and civil engineers who are tasked with inventing new types of houses every time.
- siavosh 14y agoNot that I disagree with you, but have you looked at a typical architect's salary and job prospects?
- paulbaumgart 14y agoIt seems there's a lot more demand for different types of software than there is for different types of houses.
- BerislavLopac 14y agoEspecially since most of the house design is now done by software.
- IanChiles 14y agoTo add on to your point - you can program anything with little to no startup costs anymore, whereas you can't build anything architecture wise without large amounts of money.
- VLM 14y ago"But that doesn’t explain why people in the Philippines or China aren’t training themselves en masse to be able to soak up the $30-100/hour jobs that would be readily available to them" Rentacoder is readily available, its just that word on the street was most jobs paid about $5/hr. Amazon's Mechanical Turk has some extremely fuzzy stats and not much more than anecdotes but earning around $7/hr has been reported. Higher pay, more fun, less effort? Waiter/bartender at a tourist restaurant is probably the most fun and could pay even more. Why is the market so fuzzy and illiquid? Why aren't there ten places I can go to find "write perl to import this old fashioned CSV file into that database" and see exactly what thats worth? Fix that and then wonder where the people are.(edited to rephrase there's no market, so no one plays the market)
- raverbashing 14y agoBeing a waiter at a tourist restaurant is pretty much an 'on/off' occupation. You need a small amount of specific knowledge to start working (at a specific place), and leaving the job takes almost no knowledge away. Software development is a very slow start, depending on the project. Just try and see how long until a developer has the development environment. Yes, it should be a 'turn key' solution but that's often not the case.
- redblacktree 14y agoCheck out vagrant[1]. I can have a new developer set up in about 30-60 minutes. [1] http://www.vagrantup.com/ http://www.vagrantup.com/
- npsimons 14y agoI don't think the GP was referring to development environment setup time as a significant amount of the spin-up time of a software engineer. If it is, you've got bigger issues.
- raverbashing 14y agoWell, you're assuming that 1 - All projects can be run in a VM 2 - Setting the development environment is all a new developer needs Let me try to instil some expanded vision into things: - I can't have a VM for our specific embedded hardware with its specific devices - I haven't seen a VM for AIX on Power (maybe except running on Power HW) But most importantly, Vagrant can't put on the developer's head everything specific (s)he needs to know to do the job. Like how to use the proprietary utils that we need to integrate in a project. Or what the project is all about.
- beeffective 14y agoWhy doesn't the pay reflect the supply and demand? I consider myself a decent developer with 9 years of industry experience in several domains and my salary is only $88k. It sounds like based on the demand, I should be pulling in a lot more.
- pekk 14y agoI think it's because (though it's always carefully framed to leave this implicit) the argument here really isn't whether there's a developer shortage. It's whether to do more stuff like H1B which increases supply and decreases wages.
- paulbaumgart 14y agoWhere are you located? Geography is still a big factor in hiring decisions, so demand is not evenly distributed. In Silicon Valley, $88k is what good-but-not-great fresh college graduates get paid.
- MatthewPhillips 14y agoThe point still stands, unless you work for the handful (or less) of companies that pay FU money to their engineers, you're still paid equivalent of what a Dentist makes, and much less than what a good salesperson makes in a much much smaller city. I think the answer is that the demand isn't actually very high. Programming is still considered a cost center for most companies, the quality of the product is determined by how much they are willing to spend on salary, not vice versa.
- paulbaumgart 14y agoJudging from how much attention all the good software engineers I know get from recruiters and from conversations I've had with people actively hiring, the demand in SF/Silicon Valley is incredibly high. I have personally taken advantage of this demand through lucrative contracting gigs. Sure, crappy companies in dying industries might not be able to compete, or might not even realize that they're failing to compete, but why care about them? They don't make the market for people who are actually good at software.
- cateye 14y agoBy good, it is often meant to be in the 5% of the standard normal distribution. So "good" is an infinite shifting condition. When there are more developers, it becomes just harder to be good enough. This scares therefore often novice off. They implicitly think that they have to be "really" smart to retain a life long job. I think that there are enough developers. But everybody is searching for the geniuses that can create a competitive advantage.
- sehugg 14y agoby now most of the world’s programming would be happening in low-to-medium-wage countries and that customers wouldn’t have to pay a lot for high quality software. But it hasn’t happened (e.g., see my posting about California’s state government spending $327 million on a straightforward ecommerce site Uh, the reasons government IT projects are typically huge wasteful boondoggles are not because programmer salaries are too high or programmer talent is mediocre.
- solistice 14y agoI was puzzeled over that number too, as putting up a site like that, even if you are buying the servers from scratch and hire a first rate web dev, shouldn't cost more than 10-20k$. I went through to the linked artice, and appeareantly, most of it went for consultants. I mean there are consultants out there that are worth their money, and I bet the site is kai-zen optimized very nicely, even though I can't tell, but 327 million $ is quite a hump for that website. The main problem seems to be that government contracting is not a regular market, else joe's jquerry shack would have ourrun and outpaced accenture in both cost and quality.
- gte910h 14y agoI am also guessing there is a backend component that isn't on yet? Additionally, gov't is traditionally entirely unwilling to agree to strict requirements, so require vast overbidding
- solistice 14y agoi guess there is a backend there that isn't accounted for yet. I went with 5k for web design and 15k for 3 pretty average servers (using jeff atwoods specs as a general number, where he comes out to $2,750 for raw server cost without allocation), but in retrospect, 3 average servers could be somewhat small for a site like that.
- jshen 14y ago"if you are buying the servers from scratch and hire a first rate web dev, shouldn't cost more than 10-20k$" this number is just as absurd as the $327 million.
- bengrunfeld 14y agoBecause people great and small are afraid of programming. It is the big hidden terror inside of a circus box that grown-ups use to terrify children and village idiots. Serious applications seem so complex with their millions of lines of code that people are simply scared off. I know that the industry is now trying to entice youth and professionals to begin learning code with things like http://code.org http://code.org, but they need to focus more on the rewards (i.e. like how much money you can make, and job stability even during economic crises). Learning to code is hard, and you need to commit your brain to 3 years of serious edumacation, and then 5 years of industry experience if you want it to succeed. That's a tough bill for anyone to swallow.
- tomkarlo 14y agoBecause there’s a longstanding view that since programmers love what they do, they shouldn’t be compensated as highly as other fields with demanding requirements. Employees at large tech firms are as profitable as bankers, yet they get paid a fraction as much compensation. Not to mention that it generally takes years and years of unpaid homework / hobby work to learn how to be a good software developer - just taking four years of college CS with no outside hobby work will not prepare you in any way. Nobody expects that from doctors, lawyers or bankers. So the pool of great programmers is generally limited to those who love the job _so much_ that they’re willing to forgo income to do it. Someone with a “125 IQ” and strong math abilities could get a job on Wall Street, or as an actuary, etc and make more money for less frustration. At Amazon, they target 75th percentile comp – but I really doubt they also target hiring 75th percentile developers on average. So they’re basically trying to hire 90th percentile employees for 75th percentile comp. I’d argue the entire industry does that in comparison to non-engineering industries. The next time a CEO complains they can’t hire enough engineers, ask them how much over market they’re offering.
- jskulski 14y agoI see your point but I think "forgo" is a strong word. I have made and will continue to make a great salary with great benefits and be able to choose companies based on cultural fit. That's something that I would never give up for an extra 50k a year.
- jshen 14y agoFor an extra $100k a year? I think a lot of programmers don't realize how much some other programmers get paid once you factor in other incentives outside of salary.
- cantankerous 14y agoAgree with you there. I have some friends that do the kinda dev I feel is radioactive, but then they tell me about their benefits and wages and I'm like "where's the hazmat suit?"
- 14y ago
- willholloway 14y agoThe author of this would have his answer if he understood the IQ floor necessary to be a good engineer, access to computers at a young age and willingness to troubleshoot obscure problems is relatively rare in the population. There are far less high IQ individuals in the world than the world needs right now. We can teach coding to young people, but as of now we can't reliably increase IQ. We have studies that prove agents such as piracetam administered to pregnant rats increase the intelligence of their offspring. If the Tech titans that want cheaper labor really want to solve the problem of lack of quality engineers they could think long term and throw massive amounts of funding at neuroscience and nootropic research. In the information age we all live and die by our wits. Policy and R&D money should be directed at raising the mean intelligence of the population through novel scientific breakthroughs. Far easier though to just lobby for government policies that make it easier to import the rare kind of people Google, Microsoft and Facebook need. So we have a situation where no one is focused on solving the root of the problem (a need to increase the cognitive abilities of humans) and no one is lobbying to protect the financial interests of working coders in the United States.
- jh73 14y agoAs a scientist, even mentioning IQ is a good way to lose your career in the US. It's frustrating because the general population are turning to off-label drugs and supplements. 30% of college students are taking some sort of ADHD drugs for performance enhancement and even more when including nootropics.
- willholloway 14y agoThe gigantic demand for stimulants and nootropics is a good indicator that people are feeling the strain of the information age and reaching for answers. We need far better answers. The situation is only going to intensify as more low end jobs are automated.
- danielweber 14y agoDo we know that those drugs actually help? I can easily see 30% of college students doing something because they think it helps.
- diego 14y agoFor the same reason there isn't a glut of NBA players. Anyone can play, few people can become good enough. Next question. Edit: disagree? Downvote if you want, but please counterpoint.
- jdminhbg 14y agoI didn't downvote, but programming is nothing like playing in the NBA. The NBA is zero-sum: There are by definition (15 players per team * 30 teams) NBA players in the world. There exists no such hard limit on "good programmers"; if there were more good programmers people would pay them to solve problems that currently go unsolved.
- jshen 14y agoThe NBA has a fixed number of players. The analogy doesn't fit at all. I'll bet if you made 10 new teams, and only allowed them to sign players that other teams don't want, that a number of them would be better than the worst teams in the NBA right now.
- arunabha 14y agoIts easy to run, all you need is a pair of shoes(not even that in the extreme case) and a bit of open space. I wonder why we don't have olympic runners in every neighbourhood ? More seriously, I suspect its a consequence of the 10000 hour rule applied to a discipline which builds some of the most complicated structures known to man. It takes a lot of training, formal or informal and a lot of good feedback to make a good software engineer. Plus, CS is not the easiest of disciplines to study. That, along with the fact that good practitioners, in almost any discipline are by definition the top few percentage. The bigger question would be, as the profession matures and builds up an institutional body of knowledge like civil engineering, would the number of 'good enough' engineers rise.
- vampirechicken 14y agoBecause businesses want to cut corners and ignore the good practices and disciplines of software. Then they complain that it's the software engineers' fault.
- ap22213 14y agoI think there may be a glut of potentially good software engineers. In fact, I know several high-IQ programmers who are skilled, but sadly, probably unemployable. I think the problem seems to be (at least) four-fold. 1. They don't have the prerequisite cultural background. 2. They are self-motivated to pursue their own creativity but less interested in others'. 3. They don't have a fondness for excess money. 4. They have a hard time being obedient and subordinate. Maybe controversial, but the way I see it.
- paulbaumgart 14y agoThere is absolutely a market for really good creative people who aren't obedient or subordinate. You have to be able to back up your disobedience with results, though.
- tomkarlo 14y agoBuilding "business-scale" software is almost always a team effort. It doesn't matter how brilliant someone is if they can't work well with their team; in fact it's generally a recipe for disaster. 2 and 4 here are deal breakers for any decent software team. The ability to "disagree and commit" is crucial for members of any team that want to both express creativity, but deliver results.
- zampano 14y agoWhy wouldn’t the millions of unemployed Americans train themselves to code? I'd say this is exactly what my friends and I have done. We had been nerds growing up but the majority of us went into liberal arts majors in college. After graduating and finding out that our slips of paper were meaningless, a few of us taught ourselves how to program and a few of us built upon college IT jobs to go into system administration. When we were younger, I think we generally had some pretty bad misconceptions about the software development as a whole, especially regarding relative difficulty compared to other careers. While the money and job security were the things that drove (most of) us in that direction originally, we've all become completely obsessed with a tech world that we generally misunderstood when we were younger. I've spent a lot of time trying to convince other out-of-work nerds to do what we did but I think most either don't believe you can get work without a CS degree (by assuming that we just got lucky) or think that it is well beyond their mental capacities. I'm not surprised a lot of unemployed americans are scared of trying something intimidating, but I think you'll see more and more young un(der)employed americans start learning code, especially as the learning resources become more and more available and more and more accessible.
- numo16 14y agoI don't know about how other schools approached programming careers, but I know my school tried to make it out like you had to be some crazy math genius in order to do basic programming. Compound that with the fact that my school didn't offer any programming classes for people to even see if it was something they would be interested in, and we're left with me being the lone software engineer in from my school for the past several years. They scared everyone else away.
- ExpiredLink 14y ago> Why isn’t there a glut of good software engineers? Because a career in computer programming (still) sucks: http://www.halfsigma.com/2007/03/why_a_career_in.html http://www.halfsigma.com/2007/03/why_a_career_in.html > Programming is easier to learn than a foreign language Maybe 'coding' but not programming and definitely not software engineering.
- waps 14y agoThe advantage older coders bring to the table is full-stack understanding, at least at the moment. Sure, maybe not as useful to bring your site off the ground, but MUCH more useful for any kind of optimization.
- bradleyland 14y agoThere is a tremendous difference between being able to "program" (the ability to type out syntax that will execute) and being able to design large, complex systems. Stop and think for a moment about the complexity in interaction between components in a modestly complex software product. I know plenty of people who can write code competently, but cannot design software very well.
- rayiner 14y ago1) I'd imagine programming ability is fairly correlated with IQ, which puts a limit on the percentage of the population that has the requisite analytical skills to be effective programmers. 2) Programming is as much about communication as it is about writing code. People have this idea that you can reduce a program to a precise set of specifications that can be formulated by non-programmers and shipped off to China to be coded up. I find the idea ridiculous. People don't come with to lawyers with the precise contours of the case they want to build and simply ask for the lawyer to write up the briefs. They don't come up to doctors with a precise treatment plan and simply ask them to administer the medications. Programmers are problem-solvers, and that's a two-way street, and that makes shipping critical programming work over to China and India untenable because of communications barriers.
- tomkarlo 14y ago"People have this idea that you can reduce a program to a precise set of specifications that can be formulated by non-programmers and shipped off to China to be coded up." Yup... If this was actually possible, we could just write software to do it and not need programmers at all.
- marcbarros 14y agoI"m not a software engineer, but I wish I had learned it in college instead of studying business. Now as a startup founder I'm at a disadvantage because I can't just start building products I see, I need a team just to start. As the CEO of Contour it wasn't about pay for great engineers. It was about having an interesting problem to work on and a company culture they wanted to be a part of. Yes you compete on pay, but for a start-up, everything else you offer is as important. Growing up I didn't even know what you could do with engineering. I loved math, but had no idea what you could do by being an engineer. I think a lot of kids don't realize the creativity you can harness with engineering and how you can "build" real things. It was always presented to me in terms of math and being nerdy. Just like writing, I hated it because it was about "grammar" and not about expressing yourself. Now I love to write. For kids I think it is framed up wrong and until more engineers have kids and teach their kids just what you can do, we will be short software engineers. I'm now trying to teach myself with codeacademy and it's hard.
- deleted 14y ago[deleted]
- scrame 14y agoBecause most software engineers aren't very good.
- CmonNoReg 14y ago...and they do not want to be better.
- CmonNoReg 14y ago"As Google hires all of the world’s good software engineers" - probably an exaggeration, but definitely not true. At least in UK salaries there are lagging, only good for experience and to have it on your CV. Any company is only hiring the best people they could identify of those who applied.
- taylodl 14y agoI am left to wonder how everyone could have been so wrong in predicting that the world would be glutted with good programmers and sysadmins by now I stopped reading the article at this point. Seriously. I've been programming for nearly 30 years and in that time no one ever thought there was a risk of our having a glut of good programmers. Ever. Hell, finding good talent back then was much more difficult than it is now. So stop your whining.