15 ms·

Software engineering research is a train wreck

372 points by martincmartin 5y ago

zabzonk 5y ago
If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.
- routerl 5y ago
  There's another strategy that works better than searching for a keyword, which is to read the article. It's about the difficulty/expense of systematically and scientifically studying the software development process (e.g. finding out what data actually exists to support the claim that "a bug is cheaper to squash when it is caught during the specification process"). The title expresses frustration not at science itself, but at how needlessly difficult this process is, since both the secondary and primary academic literature surrounding it are of extremely low quality.
  - AnimalMuppet 5y ago
    But specifically with respect to software engineering, it's really hard to do solid, repeatable experiments. It's not like physics, it's like sociology or psychology - which also have trouble doing precise, repeatable science. Why is it like sociology or psychology? Because it's about people, not about things. Software engineering is not just about languages and programming techniques; it's about how people interact with those languages and techniques. The people have far more variability than the languages and techniques do. Cutting through that to be able to accurately say something about any language or technique is... really hard.
    - deregulateMed 5y ago
      I think at the present time, software can be more like science than sociology. Both suffer from abstraction. I think psychology when closer to biology is similar to programming like running the programming language C experiments.
  - deregulateMed 5y ago
    Despite insistence of programmers wanting to be scientists and engineers, it's not true to the definition. Maybe safety critical C or assembly is engineering. Maybe testing code for speed can be considered science. By the time any useful program is finished, there's significantly more tradition and art than Science and engineering.
    - routerl 5y ago
      Sure, but I think the underlying instinct behind "evidence based software engineering" is precisely about turning those "maybes" into measurements. In principle, I'm in favor of this endeavor, and the article at hand expresses frustration about apparently arbitrary barriers towards accomplishing these measurements. I don't know about you, but I'm generally always in favor of "let's measure more things and build more models from that data, to make sure we're actually improving"
- coldtea 5y ago
  Try figuring it out through the context. There are paragraphs upon paragraphs of it in the article.
sseagull 5y ago
I generally agree with everything, but thought I would just use this as a springboard into a related topic: > What’s the difference between a “bug” and “defect”? Well, one’s a bug and the other’s a defect, duh! This kind of issue is common but I'm not sure how to avoid it. Any group of people > 1 will start to use their own lingo, which often is made up of similar words from "outside" but have different connotations. This is true in science, medicine, software engineering, law, everywhere. (I mean, why would I search for "bug"? Like, I'm searching for problems with computer code, not insects!) This phenomenon unfortunately leads to misunderstandings with the general public, which leads to mistrust. Part of it is on scientists (and lawyers...) to be clearer in their communication, but I think it is also on the public to recognize that when reading scientific literature they are not the intended audience and are therefore missing a ton of context that is not explicitly stated. Also: >I’m sure this is slightly easier if you’re deeply embedded in academia Also depends on the field. Chemistry has SciFinder, which although very expensive for institutions, is very good. It is fairly specific to chemistry though (and some overlapping fields). His process for finding node papers and grinding through citations is pretty much how most scientists do it, though. And conferences.
dogorman 5y ago
> I’ve checked a few other papers and think I’m tentatively confident in this line of reasoning: certain bugs take more time to fix (and cause more damage) than others, and said bugs tend to be issues in the design. I haven’t vetted these papers to see if they don’t make major mistakes, though. It’s more time-consuming to do that when you’re trying to synthesize an stance based on lots of indirect pieces of evidence. You’re using a lot more papers but doing a lot less with each one, so the cost gets higher. Empiricism and quantitative metrics have indispensable value, that much should be clear to everybody I hope. But too often people forget (or have active contempt for) the value of qualitative metrics which can only be judged subjectively. Such considerations are naturally harder to deal with than cold hard data, so it doesn't surprise me that people want to avoid it. But when you blind yourself to the qualitative and subjective, you do yourself a huge disservice. Just ask McNamara; he thought he could win the Vietnam War with quantitative metrics and utterly neglected difficult to quantify metrics like public sentiment, both in Vietnam and America. I see echos of this in our industry today; we love to talk about empirical measures like the number of bugs, but subjective metrics, like the severity of those bugs, receive less attention. Many university programs are set up to address this, by making engineering students earn credits in the humanities as well. But I fear the value of this is often inadequately explained. Contempt for the humanities and scientism go hand-in-hand, and are a worrying trend particularly in the tech industry.
- rmetzler 5y ago
  > But too often people forget [...] the value of qualitative metrics which can only be judged subjectively. When I read the rant, I remembered a story about my university days and your sentence prompted my to write it down. 10 years ago I had to write a meta paper about Test Driven Development (TDD). I was researching this topic and found studies and other meta papers. Some of the studies were from the likes of Microsoft where they explained they wrote two different drivers with comparable lines of code - one with TDD and one without - and tracked how many bugs they found after version 1.0 and how fast the driver projects were delivered. So in these non-trivial multi-month multi-person projects they claimed TDD was extremely useful to reduce bugs. Other papers had 20 students learn how to program with TDD and another 20 students without and tried to find differences there. They couldn't find too many differences in the groups doing something like a 2 hour project. Since the group who just learned TDD was a little bit slower, they concluded neutral or negative value of TDD. When I looked into the meta papers comparing these, I found some that were judging both of these papers as being of equivalent value, which to this day I'm really wondering about.
SebastianFish 5y ago
Testing different methods of development in terms of speed, cost and quality is really hard. The most convincing approach to me would be a single blind experiment to hire two software development teams and have them build to the same set of requirements in two different ways. But then it is hard to know whether you are really comparing the method of software development or the quality of the software teams. So two software teams isn't enough to get a statistically valid inference. You can see that, given software development rates, this could become a very expensive experiment. Last point. I think that even writing a specification down to the level that it could be implemented using formal methods might be the biggest game changer. Agile stories rarely come even close to covering all of the potential edge cases. If we had a process that required product owners to literally think through all possible failure modes (what systems of formal methods do) and write out how to handle them then the cost of writing specifications would go way up. Per economics, I think we would end-up with simpler specifications which might be its own benefit.
- nradov 5y ago
  Requirements analysis is an integral part of the development process. Giving teams fully specified requirements at the beginning of the experiment wouldn't be realistic.
  - AnimalMuppet 5y ago
    Depends. You could then run experiments on team performance in the requirements-to-code phase, separate from the generating-requirements phase. That has its place. And then you could experiment with teams trying to convert informal requirements to formal requirements. That might let you learn some things about the parts that you couldn't learn if you dealt with the whole.
    - nradov 5y ago
      Those are not separate phases, they take place simultaneously. No one does waterfall development anymore.
Bayart 5y ago
It's completely besides the point, but the capitalizing of the I of « **Ing » in the title makes me unreasonably grumpy.
- deleted 5y ago
  [deleted]
jeffreyrogers 5y ago
I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough, temperature variation causes parts to expand by different amounts, parts not round or flat enough, etc.). Once we know what errors we need to control and their rough order of importance we can start improving the machine design to control them better. In theory, something similar could be done in software engineering (formal methods are part of this, but not a full solution). Rather than an error budget, you'd have some sort of bug budget, where you tracked what sort of bugs were caused by what sort of constructs, and design your program in such a way to minimize their chance of being introduced. I've never heard of anyone except Dan Bernstein[1] actually doing anything approximating this. Probably because the perceived level of effort is too high. I actually don't think it would take that much effort, but it would require quite a bit of organization to track where bugs are introduced and what their root causes are. This is probably why Bernstein, an individual, is able to do this, while no large team (that I'm aware of) has done anything similar. Of course, just like your toaster doesn't need to use precision machine design techniques (an over-engineered toaster is a bad toaster), most software doesn't need the effort/expense of a rigorous design process either, but some would benefit from it. [0]: https://dspace.mit.edu/handle/1721.1/9414 https://dspace.mit.edu/handle/1721.1/9414 [1]: https://cr.yp.to/djb.html https://cr.yp.to/djb.html
- deleted 5y ago
  [deleted]
- ska 5y ago
  > while no large team (that I'm aware of) has done anything similar. In areas like avionics, medical devices etc., there are formal production engineering methods applied to software as a matter of course. It can very work well, but it is definitely expensive compared to industry average.
  - JimTheMan 5y ago
Helmut10001 5y ago
Wow, thanks for the link to sci-hub.st - this is awesome! I can finally access Elsevier's "walled" content again.
- teddyh 5y ago
  https://whereisscihub.now.sh/ https://whereisscihub.now.sh/ Currently redirects to: https://whereisscihub.vercel.app/ https://whereisscihub.vercel.app/
  - Vinnl 5y ago
    Oh hey, I made that, but it was taken down by what was then called Zeit. Looks like that didn't persist through their Vercel rename. Note that it just fetches the current URL from Wikidata.
JohnFen 5y ago
It sounds like the author doesn't hate science (which is a nonsensical thing to hate), but academia. Those are two different things.
lazyant 5y ago
I don't know what "FM" and other initializations are. Please when writing an article, expand them in parenthesis the first time they are mentioned.
- raldi 5y ago
  It’s expanded the first time it’s used (in the sentence preceding the one with the abbreviation).
- cpeterso 5y ago
  FM = Formal Methods
- MiscIdeaMaker99 5y ago
  I didn't catch that either. Sucks you're getting downvoted for it when it isn't as obvious as others seem to think.
  - hwayne 5y ago
    I'll put a quick link in
IshKebab 5y ago
This tallies with my experience. It isn't early bugs that are expensive to fix. It's early design mistakes. For example choosing the wrong language, framework or architecture. Dropbox using Python is a good example. Or Python's GIL.
- klysm 5y ago
  They aren’t really exclusive at all. Early design mistakes can also be really expensive, but a really scary bug that nukes all your data could also suck.
  - IshKebab 5y ago
    They mean the cost of fixing the defect, not the severity of bugs.
- foobiekr 5y ago
  The GIL is a good example because it removed optionality from Python by having its semantics leak out. Using a poorly performing tool is a different problem entirely. Most of the really horrible errors in SW design involve choices where the semantics above will be visible and encoded as a subtle (or not so subtle) dependency that cannot later be fixed without an infeasible level of effort. I deal with these regularly and they are the true original sin in most sizable software projects.
MengerSponge 5y ago
Sci-hub is great, but it should be your last resort. University librarians have explained this to me, but it has to do with how Sci-hub counts towards access counts of obscure journals. Unpaywall provides an excellent browser extension. https://unpaywall.org/ https://unpaywall.org/ If you have time, you can request a copy of a paper directly from the corresponding author. We love for people to be interested in our work, and we would rather you put that $40 access fee towards something that benefits society.
- arkitaip 5y ago
  Amazing extension. Thank you.
mkl95 5y ago
A lot of software in production is designed like a bunch of railroads that are unnecessarily twisted and dangerously run into each other for no apparent reason. So train wrecks are to be expected.
ftio 5y ago
I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about areas of software engineering research outside of productivity.) Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. It does not have natural, universal laws. What makes a team 'productive' varies massively based on constraints imposed by business model, product, customer expectations, leadership values, and of course the individuals of which it consists. I do not believe in the possibility of a "General Theory of Productivity." I'm highly skeptical of attempts to quantify the precise relationship between error discovery stage and cost in a way that is generalizable, although I think it might be possible given a large group of engineers using a highly homogenous process, tools, and accounting. Google is pretty close to this (common dev infrastructure across tens of thousands of engineers), and even across Google this kind of generalization would be extremely difficult. There is no universal physics of software engineering. As a result, academic research into productivity can be difficult to generalize (which is why, I think, you often see researchers twisting themselves into knots). Instead, my rec is to focus on a few key metrics that are aligned with your business' or team's goals (search for DORA for a good starting set) and to reflect often on what you feel makes your team work well and what doesn't.
- shusson 5y ago
  > Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. Do you think software engineering will remain a creative endeavour in the future? We've only been programming in high level languages for 60-70 years. All the while Moore's law has been in effect.
  - spaetzleesser 5y ago
    “Do you think software engineering will remain a creative endeavour in the future?” I would think so. If something can be automated or be done with a reproducible process it will get automated or performed by lower paid people. But I can easily see how software and automated processes will reach a point of “good enough” for most purposes sometime in the future and there will be much less software engineers doing creative work. For example I would expect a lot of repetitive front end backend work will go away.
kragen 5y ago
"View Source" may be helpful if you're just seeing a blank page.
truenindb 5y ago
this problem where published science is often wrong and you don't need to actually be an engineer that history will not record the name of is completely new, and also, airplanes are impossible and where actually invented by The Wright Bros, because they had the Wright stuff, right?
shusson 5y ago
> ...“bugs found in requirements are 100x cheaper than bugs found in implementations.” ... There’s one tiny problem with the IBM Systems Sciences Institute study: it doesn’t exist It's funny to think how much of an effect this chart had on software engineering as a whole. I remember learning it at university and until today I thought it had some basis in science.
- eschneider 5y ago
  It "feels right" and is therefore "truthy."
  - krisoft 5y ago
    Do you guys doubt that it is right? What kind of information would you require to show you that it is true? I'm one of the lucky ones who tends to work on safety critical "complex cyber-physical systems". Is that maybe the difference? There is the case where a developer runs unit tests before a commit. Some of the tests are unhappy. Developer fixes it and commits it up. Maybe hours spent? Then there is the case where something goes funky in the real-world all-up integrated system at the test range. Even in the best case tens of people waste a day. If we are unlucky it is a hard one and a small army of the most senior developers with the best operators and hardware people are hunting the bug for weeks. I fear to even sum op the wasted cumulative work days. I lived through multiple of these at different companies on different projects. (Both the first kind and the second kind.) Is the question if this is true? Or maybe the question is if this is true for your area of the industry too? Obviously I won't be able to provide sources for the ones I worked on. I bet no company really would want to release the hard data on these things. So let's look at some well-publicised problem caught in production: The Boeing 737 MAX MCAS issue. 346 deaths, 1 year, 8 months and 5 days grounding and the estimated direct costs are US$20 billion. Obviously this is all super anecdotal. I just want to understand what part of the question/problem you have doubts about.
    - twobitshifter 5y ago
      I think that’s the point - the chart confirm your bias and everyone else’s so we accept the results at face value. Nobody really knows if those bar heights were ever measured but of the millions of people to see that chart few questioned it.
qznc 5y ago
Greg Wilson is more positive. Though he does agree that nobody cares. https://third-bit.com/2021/07/17/software-engineerings-greatest-hits/ https://third-bit.com/2021/07/17/software-engineerings-great...
- dang 5y ago
  That (or more likely https://www.youtube.com/watch?v=HrVtA-ue-x0 https://www.youtube.com/watch?v=HrVtA-ue-x0 since that's where the info is) would make a good HN submission in its own right. Probably best to wait a few days to let the hivemind caches clear (followup/copycat posts aren't great: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=follow-up%20by%3Adang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). If you submit it, let us know at hn@ycombinator.com so we can put it in the second-chance pool (https://news.ycombinator.com/pool https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308), so it will get a random placement on HN's front page.
  - hwayne 5y ago
    This is a cool system!
- jdlshore 5y ago
  I'm skeptical of Wilson's reporting. One of his quoted studies looked interesting (slide 20, Fucci 2016, "An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach"), so I looked it up. Wilson describes that study as involving 39 professionals on real projects. But when I looked at the actual study [1], it involved 21 graduate students working on two toy problems, the Bowling Game kata and "Mars Rover API." This is a disappointing misrepresentation of the study. A charitable interpretation is that he was referring to a different study, and accidentally put the wrong reference on that slide. Or maybe the actual talk explains the difference. I'm not sure—I didn't look deeper. [1] http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf
deleted 5y ago
[deleted]
blunte 5y ago
> But first you gotta find the primary sources. Yeah, and the real primary sources are the companies building and running software. They don't have time or interest in the long term research. And likely they aren't even concerned with their own technical situation 12 months in advance. They just need to "deliver", they think. So unless you can find some clever way to gather all their data (Sentry, AppSignal, ...?), you will never know what's really happening in most places. Your research will be artificial without that real world data.
lifeisstillgood 5y ago
My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. Software is more like a novel than a bridge. Maybe, maybe we can apply engineering terms to machine code or assembly. But the abstract levels we all work at - not a hope. I mean this is a good thing. It changes how we think about software - engineering has changed the world - and it's in the hands of a few professionals. Imagine software in the hands of ... everyone. It's like Eric Idle gather mud in the Dark Ages imagining what a literate world that reads the New york times will be like
- hwayne 5y ago
  > My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. 1. This is false. One of the reasons making bridges is so expensive is because each bridge is a precious special snowflake with lots of challenges specific to that bridge project and no other bridges. 2. Bridge building is a very small part of civil engineering, and civil engineering is a very small part of traditional engineering. Maybe software is more like process engineering, or chipset design, or subsystem integration, or workflow optimization.
  - lifeisstillgood 5y ago
    1. I agree bridges have special challenges, but ... well for 2,000 years there was a keystone, and an arch. Yes the soil and the land were challenging, but I find it hard to argue that every stone bridge was a special snowflake. Snap a supporting cable on any suspension bridge and they all have the same failure mode. I think we are stretching analogies - is "build a CRM for the sales department" simple as in "there is a keystone data design common across all CRMS - and all the rest is "shifting sands of the river are same as shifting alliances in the boardroom" I just try to keep this simple. Coding uses symbolic representation to describe a model - this is what language is. Compilers do / can do things that simply don't exist conceptually in Engineering. Coder manipulate symbols. It really is a language thing. 2. yes. I was just using bridges as an example of engineering. As CRUD websites are an example of software engineering.
Quanttek 5y ago
> The average developer thinks empirical software engineering is a waste of time. How can you possibly study something as complex as software engineering?! You’ve got different languages and projects and teams and experience levels and problem domains and constraints and timelines and everything else. Why should they believe your giant incoherent mess of sadness over their personal experience or their favorite speaker’s logical arguments? This argument could be used for literally any science but, for some reason, it seems to fall on attentive ears mostly in the software industry.
- hwayne 5y ago
  I think it's common in a lot of fields. It's def prevalent in medicine, for example: people are skeptical of any research on doctor behavior.
- twobitshifter 5y ago
  I think the issue is that management is not done scientifically. You can see this with something as estimating for a project. Higher ups just ask “how many hours and when can you get it done” - they don’t want you to go through past similar work review it’s timeliness and hours, divide it into smaller tasks, build a project schedule, review productivity trends and resource loadings, or any scientific way of arriving at the number. What they really want is a seat of the pants WAG, which isn’t too big to scare the client, that they can use to get going, with the understanding that shit happens along the way.
cycomanic 5y ago
I found the complains about how reading papers, do literature research, evaluating papers and the field etc., is difficult somewhat amusing. That's actually what a large portion of the PhD education and becoming an academic is about. Yes, it is not easy, if it was a PhD wouldn't take between 3 and 6 years. Generally, the author makes some good points and there is definitely often a disconnect between academia and industry. I think it is important to remember that science is much more like a directed random walk toward an unknown goal. If industry wants specific answers to specific questions they could (should) finance the studies that provide them the answers. However, my impression is that when industry finances significant studies it is much more toward validating/confirming their already established practices, products... (the topic of these "bought" studies is another can of worms in the topics of academia and industry interactions)
mcguire 5y ago
"Here’s the only technique I’ve found that works, which I call scrobbling even though that means something totally different: "1. Search seed terms you know, like “cost of bugs”, in an appropriate journal (here’s one). "2. Find papers that look kinda relevant, skim their abstracts and conclusions. "3. Make a list of all the papers that either cite or are cited by these papers and repeat. "4. Find more useful terms and repeat." Congratulations! You're a grad student now! P.S. When I graduated, I had a filing cabinet with two drawers full of papers that felt important enough to save printed copies, from just this process. A couple of months ago, I carted them out to the burn pile, threw them on the pile of leaves and brush, and held a Viking funeral. And I didn't even feel bad. :-)
- hwayne 5y ago
  I got into software specifically because I didn't want to go to physics grad school, and now I read more papers than I ever did in physics. Man plans, God laughs.
- titzer 5y ago
  My impression on reading this list was, uh yeah, there's no avoiding that. Except if you know an expert. Then just ask an expert. Search engines for papers suck, bad. At the fringes of human knowledge, you need an actual brain to process cutting-edge research. Either your own non-expert brain, or an expert brain to help guide you. That process is the same in every field. Older fields have textbooks with knowledge distilled from experts. That's a good place to start to get the state of the art > 20 years ago.
  - frickinLasers 5y ago
    | Search engines for papers suck, bad. This screams the need for an intelligent search engine, with summarizing features that point out relevant parts of the paper, and (imho) a UI and algorithms that reach across disciplines and encourage discovery. So, for instance (I'm making this up), someone looking for a way to understand interferometry data might stumble on a useful regularization technique from image processing. I imagine academic libraries would pay really good money for that. Of course, it would help if a mostly complete body of scientific literature was available to crawl--like SciHub.
TeeMassive 5y ago
The only truths that I've known so far to be true: 1) Small patches with quick feedback 2) Automate the manual stuff, even if you're not doing it often you'll probably like having that script ready and tested when you'll have to do it again in one year.
walshemj 5y ago
The Bug Vs Defect made me laugh back at my first job we partnered with the BSI to do research that went into BS5750 / ISO 9000. The Head of software side at the BSI commented that tat for old timers in QA - software was odd as they "did not know wear to hang the defect tags"
aranchelk 5y ago
> The average developer thinks empirical software engineering is a waste of time. Bold claim. Any data at all to back that up? My guess based on experience would be most developers are unfamiliar with the term and haven’t made a judgement one way or the other.
aeturnum 5y ago
Software engineering research is immature and, as such, has few (no?) strongly-supported claims that the entire field relies on. I agree this is a frustrating state of the field, but the way this is written makes the author appear to be unfamiliar with the state of things. For example: >Did I mention that all three of those papers use different definitions of “defect”? This kind of thing is really common in academia. I do not know software engineering research in detail, but there are older and more basic definitional fights in the social sciences. Defining a defect, as I'm sure the author of the blog would agree, is not a trivial decision. In general, this piece contains (perhaps hyperbolic?) dismissals of people trying to do work. It doesn't seem to respect what I imagine is earnest effort. Taken all together, it seems like the author either thinks the entire field is charlatans (which I doubt as they use the fields' conclusions at the end) or is making light of people working on problems they themselves do not have answers to in order to have a more lively writing style. It was fun to read and left a bad taste in my mouth.
- BadInformatics 5y ago
  As mentioned in TFA, the author has both written and spoken about software engineering research and related topics in some depth before. Though fiery, I think it does its job as a more informal cautionary tale on accepting research in the field without a critical gaze. Analyses like [1] and [2] (from the same author) are extraordinarily valuable because they help us wade through the firehose of published studies to find what (if any) conclusions may be gleaned. Without such a filter, we are fated to a) repeat work by slogging through the torrent ourselves, or b) taking stuff at face value. Certainly there is no shortage of the latter happening on forums like this, and I wonder how many misinformed decisions are made as a result. [1] https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ [2] https://www.hillelwayne.com/post/are-we-really-engineers/ https://www.hillelwayne.com/post/are-we-really-engineers/
Joeri 5y ago
I went into this software engineering productivity research rabbit hole a decade ago and came away similarly depressed with the lack of rigor in our field. What I’ve come to believe: - Software engineering is not an engineering discipline, it is a craft. It may or may not one day become an engineering discipline, I find this impossible to tell. - The difference between a feature and a bug is largely one of semantics, not of substance. A defect is a mismatch between expected behavior and actual behavior. A bug is when the expectation was that of the programmer that wrote the code. A feature request is when the expectation was that of a user. There is no real difference for the user between a missing feature and a broken feature. Hence why we so often get side-tracked into fruitless “bug or feature” discussions and why users don’t care when we say it is by design. - Errors will happen. People are flawed and produce errors at a more or less set average rate for each person (unless they’re tired or enjoying a ballmer peak). Errors in every step of the process, from design to deployment, are inevitable. We can however catch these inevitable errors, with automated error detection tools, or by having a fresh set of eyes look at the work. The consequence is that making high quality software necessitates taking a look at the whole process and introducing error detection steps in every part of it, whether they be automated like linters and unit tests, or human like design and code review or mentoring and pair programming. - Errors in design are more costly because they cause entire features to need to be written or rewritten, which is the most costly kind of defect to resolve. We do not need research to know this to be true. We solve this by having short feedback loops with real users, which is the only real form of agile development.
- goodpoint 5y ago
  > Software engineering is not an engineering discipline, it is a craft Software engineering is *still not treated* as an engineering discipline, it is *still treated as* a craft
john579 5y ago
Bugs in software are difficult to qualify, both in detection and resolution. Find a bug, fix it. Can't fix it? Change the specs or call it a feature. Otherwise call the program obsolete and rewrite it. For example, a new technology such as GPGPU comes out. Now programs can run much faster. This changes the problem, the approach, the implementation, the design, the requirements, and the solution. Perhaps nothing was wrong with the original CPU code, but the GPU introduced slowness bugs into the CPU code. What's the cost of fixing that bug? The cost of redesigning the problem, rewriting the code, and writing a GPU debugger so you can fix those new bugs. New technology brings change. Your faster solution brought about social change, which caused rioting, then nuclear holocaust, then the power goes out, and the whole process repeats. Add up the value of the entire world, civilization, and history, and that's the cost of fixing that bug. That's software engineering research.
bob1029 5y ago
Software engineering can only exist with an incredible amount of discipline. We routinely start new implementations in excel and perform formal normalization checks over models before writing any code. You can get away without having a strong type system as long as your tables and relations are clean. This stuff is so powerful. Figuring out how to funnel your domain instances into a SQL db and writing all your complex logic in terms of basic queries is how you can build an ecosystem in which determinism and formal proofs of correctness are possible. For me, software engineering is about correctness for the business end. There is certainly still a layer of craftsmanship below the engineering realm which enables all the magic. This is where things like performance and reliability live. Interacting with the outside world is an icky thing, and the specific implementation always depends on the use case.
- feoren 5y ago
  Software engineering is so weird -- what you've just described sounds awful to me. I believe almost the exact opposite of everything you've said. Yet I have no doubt* you've seen good results from your approach, and I get good results from mine. It's surprising how personal software engineering really is. * I do actually have doubts, because 99.9% of software is utter garbage. But I'm trying to be more positive and the point stands whether you're in the 99.9% or the 0.1%.
  - bob1029 5y ago
    Willingness to tolerate formal methods does not mean you always have to endure pain and suffering. On the contrary, you may find that you are shipping software that is so good you begin to see the value. For a lot of shops, the problem domain probably isn't complex enough to justify intense formality in the process (i.e. engineering). Once you are dealing with monsters like supply chain or factory automation, you can't play games anymore.
    - Epa095 5y ago
      > On the contrary, you may find that you are shipping software that is so good you begin to see the value. You may also find that when you finally do ship, everyone had started using your competitor, or that your customers needed something slightly different, or that the cost of waiting for you was so high it would have been better having something early. I have great respect for formal verified software, I have even touched on it a bit during my PhD, and there are definitely places for where it makes sense. Same for formal methods in general. But there are other software tasks where it is not just 'not needed' but it is actually the wrong choice. It depends a lot on the opportunity cost of delivering later, and on how well defined the tasks are, and how preffesional the customer is. And the tasks where less formal methods are the right way are not "playing games", they are solving the problem under a different set of constrains, also doing software engineering.
lincpa 5y ago
The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model Follower's success stories: Apple M1, HPE Cray SmartSim Come soon: Qualcomm, MS, Intel, AMD, Samsung, Cray, etc. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow
tdeck 5y ago
Somewhat related: I enjoyed reading the eBook "The Leprechauns of Software Engineering" by Laurent Bossavit [1]. The author tried to track down the source research behind various pieces of received wisdom. I remember that the claimed cost of fixing bugs at a later stage was one of them. [1]: https://leanpub.com/leprechauns https://leanpub.com/leprechauns
BerislavLopac 5y ago
https://news.ycombinator.com/item?id=27891102 https://news.ycombinator.com/item?id=27891102

372 points by martincmartin 5y ago

If you search for "science" in this article, you will have great difficulty in figuring out what the title means, unless you are smarter than me, which is entirely possible.

routerl 5y ago
There's another strategy that works better than searching for a keyword, which is to read the article. It's about the difficulty/expense of systematically and scientifically studying the software development process (e.g. finding out what data actually exists to support the claim that "a bug is cheaper to squash when it is caught during the specification process"). The title expresses frustration not at science itself, but at how needlessly difficult this process is, since both the secondary and primary academic literature surrounding it are of extremely low quality.
- AnimalMuppet 5y ago
  But specifically with respect to software engineering, it's really hard to do solid, repeatable experiments. It's not like physics, it's like sociology or psychology - which also have trouble doing precise, repeatable science. Why is it like sociology or psychology? Because it's about people, not about things. Software engineering is not just about languages and programming techniques; it's about how people interact with those languages and techniques. The people have far more variability than the languages and techniques do. Cutting through that to be able to accurately say something about any language or technique is... really hard.
  - deregulateMed 5y ago
    I think at the present time, software can be more like science than sociology. Both suffer from abstraction. I think psychology when closer to biology is similar to programming like running the programming language C experiments.
- deregulateMed 5y ago
  Despite insistence of programmers wanting to be scientists and engineers, it's not true to the definition. Maybe safety critical C or assembly is engineering. Maybe testing code for speed can be considered science. By the time any useful program is finished, there's significantly more tradition and art than Science and engineering.
  - routerl 5y ago
    Sure, but I think the underlying instinct behind "evidence based software engineering" is precisely about turning those "maybes" into measurements. In principle, I'm in favor of this endeavor, and the article at hand expresses frustration about apparently arbitrary barriers towards accomplishing these measurements. I don't know about you, but I'm generally always in favor of "let's measure more things and build more models from that data, to make sure we're actually improving"
coldtea 5y ago
Try figuring it out through the context. There are paragraphs upon paragraphs of it in the article.

sseagull 5y ago

I generally agree with everything, but thought I would just use this as a springboard into a related topic: > What’s the difference between a “bug” and “defect”? Well, one’s a bug and the other’s a defect, duh! This kind of issue is common but I'm not sure how to avoid it. Any group of people > 1 will start to use their own lingo, which often is made up of similar words from "outside" but have different connotations. This is true in science, medicine, software engineering, law, everywhere. (I mean, why would I search for "bug"? Like, I'm searching for problems with computer code, not insects!) This phenomenon unfortunately leads to misunderstandings with the general public, which leads to mistrust. Part of it is on scientists (and lawyers...) to be clearer in their communication, but I think it is also on the public to recognize that when reading scientific literature they are not the intended audience and are therefore missing a ton of context that is not explicitly stated. Also: >I’m sure this is slightly easier if you’re deeply embedded in academia Also depends on the field. Chemistry has SciFinder, which although very expensive for institutions, is very good. It is fairly specific to chemistry though (and some overlapping fields). His process for finding node papers and grinding through citations is pretty much how most scientists do it, though. And conferences.

dogorman 5y ago

> I’ve checked a few other papers and think I’m tentatively confident in this line of reasoning: certain bugs take more time to fix (and cause more damage) than others, and said bugs tend to be issues in the design. I haven’t vetted these papers to see if they don’t make major mistakes, though. It’s more time-consuming to do that when you’re trying to synthesize an stance based on lots of indirect pieces of evidence. You’re using a lot more papers but doing a lot less with each one, so the cost gets higher. Empiricism and quantitative metrics have indispensable value, that much should be clear to everybody I hope. But too often people forget (or have active contempt for) the value of qualitative metrics which can only be judged subjectively. Such considerations are naturally harder to deal with than cold hard data, so it doesn't surprise me that people want to avoid it. But when you blind yourself to the qualitative and subjective, you do yourself a huge disservice. Just ask McNamara; he thought he could win the Vietnam War with quantitative metrics and utterly neglected difficult to quantify metrics like public sentiment, both in Vietnam and America. I see echos of this in our industry today; we love to talk about empirical measures like the number of bugs, but subjective metrics, like the severity of those bugs, receive less attention. Many university programs are set up to address this, by making engineering students earn credits in the humanities as well. But I fear the value of this is often inadequately explained. Contempt for the humanities and scientism go hand-in-hand, and are a worrying trend particularly in the tech industry.

rmetzler 5y ago
> But too often people forget [...] the value of qualitative metrics which can only be judged subjectively. When I read the rant, I remembered a story about my university days and your sentence prompted my to write it down. 10 years ago I had to write a meta paper about Test Driven Development (TDD). I was researching this topic and found studies and other meta papers. Some of the studies were from the likes of Microsoft where they explained they wrote two different drivers with comparable lines of code - one with TDD and one without - and tracked how many bugs they found after version 1.0 and how fast the driver projects were delivered. So in these non-trivial multi-month multi-person projects they claimed TDD was extremely useful to reduce bugs. Other papers had 20 students learn how to program with TDD and another 20 students without and tried to find differences there. They couldn't find too many differences in the groups doing something like a 2 hour project. Since the group who just learned TDD was a little bit slower, they concluded neutral or negative value of TDD. When I looked into the meta papers comparing these, I found some that were judging both of these papers as being of equivalent value, which to this day I'm really wondering about.

SebastianFish 5y ago

Testing different methods of development in terms of speed, cost and quality is really hard. The most convincing approach to me would be a single blind experiment to hire two software development teams and have them build to the same set of requirements in two different ways. But then it is hard to know whether you are really comparing the method of software development or the quality of the software teams. So two software teams isn't enough to get a statistically valid inference. You can see that, given software development rates, this could become a very expensive experiment. Last point. I think that even writing a specification down to the level that it could be implemented using formal methods might be the biggest game changer. Agile stories rarely come even close to covering all of the potential edge cases. If we had a process that required product owners to literally think through all possible failure modes (what systems of formal methods do) and write out how to handle them then the cost of writing specifications would go way up. Per economics, I think we would end-up with simpler specifications which might be its own benefit.

nradov 5y ago
Requirements analysis is an integral part of the development process. Giving teams fully specified requirements at the beginning of the experiment wouldn't be realistic.
- AnimalMuppet 5y ago
  Depends. You could then run experiments on team performance in the requirements-to-code phase, separate from the generating-requirements phase. That has its place. And then you could experiment with teams trying to convert informal requirements to formal requirements. That might let you learn some things about the parts that you couldn't learn if you dealt with the whole.
  - nradov 5y ago
    Those are not separate phases, they take place simultaneously. No one does waterfall development anymore.

Bayart 5y ago

It's completely besides the point, but the capitalizing of the I of « **Ing » in the title makes me unreasonably grumpy.

deleted 5y ago
[deleted]

jeffreyrogers 5y ago

I was reading this thesis the other day[0], which is on precision machine design. It got me comparing precision machine design to software engineering. A big part of why we're able to design extremely precise machines (the author worked on a lathe used for machining optical parts for the National Ignition Facility) is because we can characterize exactly where errors will come from (e.g. structure isn't rigid enough, temperature variation causes parts to expand by different amounts, parts not round or flat enough, etc.). Once we know what errors we need to control and their rough order of importance we can start improving the machine design to control them better. In theory, something similar could be done in software engineering (formal methods are part of this, but not a full solution). Rather than an error budget, you'd have some sort of bug budget, where you tracked what sort of bugs were caused by what sort of constructs, and design your program in such a way to minimize their chance of being introduced. I've never heard of anyone except Dan Bernstein[1] actually doing anything approximating this. Probably because the perceived level of effort is too high. I actually don't think it would take that much effort, but it would require quite a bit of organization to track where bugs are introduced and what their root causes are. This is probably why Bernstein, an individual, is able to do this, while no large team (that I'm aware of) has done anything similar. Of course, just like your toaster doesn't need to use precision machine design techniques (an over-engineered toaster is a bad toaster), most software doesn't need the effort/expense of a rigorous design process either, but some would benefit from it. [0]: https://dspace.mit.edu/handle/1721.1/9414 https://dspace.mit.edu/handle/1721.1/9414 [1]: https://cr.yp.to/djb.html https://cr.yp.to/djb.html

deleted 5y ago
[deleted]
ska 5y ago
> while no large team (that I'm aware of) has done anything similar. In areas like avionics, medical devices etc., there are formal production engineering methods applied to software as a matter of course. It can very work well, but it is definitely expensive compared to industry average.
- JimTheMan 5y ago

Helmut10001 5y ago

Wow, thanks for the link to sci-hub.st - this is awesome! I can finally access Elsevier's "walled" content again.

teddyh 5y ago
https://whereisscihub.now.sh/ https://whereisscihub.now.sh/ Currently redirects to: https://whereisscihub.vercel.app/ https://whereisscihub.vercel.app/
- Vinnl 5y ago
  Oh hey, I made that, but it was taken down by what was then called Zeit. Looks like that didn't persist through their Vercel rename. Note that it just fetches the current URL from Wikidata.

JohnFen 5y ago

It sounds like the author doesn't hate science (which is a nonsensical thing to hate), but academia. Those are two different things.

lazyant 5y ago

I don't know what "FM" and other initializations are. Please when writing an article, expand them in parenthesis the first time they are mentioned.

raldi 5y ago
It’s expanded the first time it’s used (in the sentence preceding the one with the abbreviation).
cpeterso 5y ago
FM = Formal Methods
MiscIdeaMaker99 5y ago
I didn't catch that either. Sucks you're getting downvoted for it when it isn't as obvious as others seem to think.
- hwayne 5y ago
  I'll put a quick link in

IshKebab 5y ago

This tallies with my experience. It isn't early bugs that are expensive to fix. It's early design mistakes. For example choosing the wrong language, framework or architecture. Dropbox using Python is a good example. Or Python's GIL.

klysm 5y ago
They aren’t really exclusive at all. Early design mistakes can also be really expensive, but a really scary bug that nukes all your data could also suck.
- IshKebab 5y ago
  They mean the cost of fixing the defect, not the severity of bugs.
foobiekr 5y ago
The GIL is a good example because it removed optionality from Python by having its semantics leak out. Using a poorly performing tool is a different problem entirely. Most of the really horrible errors in SW design involve choices where the semantics above will be visible and encoded as a subtle (or not so subtle) dependency that cannot later be fixed without an infeasible level of effort. I deal with these regularly and they are the true original sin in most sizable software projects.

MengerSponge 5y ago

Sci-hub is great, but it should be your last resort. University librarians have explained this to me, but it has to do with how Sci-hub counts towards access counts of obscure journals. Unpaywall provides an excellent browser extension. https://unpaywall.org/ https://unpaywall.org/ If you have time, you can request a copy of a paper directly from the corresponding author. We love for people to be interested in our work, and we would rather you put that $40 access fee towards something that benefits society.

arkitaip 5y ago
Amazing extension. Thank you.

mkl95 5y ago

A lot of software in production is designed like a bunch of railroads that are unnecessarily twisted and dangerously run into each other for no apparent reason. So train wrecks are to be expected.

ftio 5y ago

I worked on engineering productivity research and measurement at Google for two years until about a month ago. (Opinions my own.) Compared to my former colleagues, I'm an infant in this area, so take this with a heaping of salt. In general, I think the author's cynicism about productivity research is justified, but I think it could have been directed more productively. (NB: the following comments say nothing about areas of software engineering research outside of productivity.) Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. It does not have natural, universal laws. What makes a team 'productive' varies massively based on constraints imposed by business model, product, customer expectations, leadership values, and of course the individuals of which it consists. I do not believe in the possibility of a "General Theory of Productivity." I'm highly skeptical of attempts to quantify the precise relationship between error discovery stage and cost in a way that is generalizable, although I think it might be possible given a large group of engineers using a highly homogenous process, tools, and accounting. Google is pretty close to this (common dev infrastructure across tens of thousands of engineers), and even across Google this kind of generalization would be extremely difficult. There is no universal physics of software engineering. As a result, academic research into productivity can be difficult to generalize (which is why, I think, you often see researchers twisting themselves into knots). Instead, my rec is to focus on a few key metrics that are aligned with your business' or team's goals (search for DORA for a good starting set) and to reflect often on what you feel makes your team work well and what doesn't.

shusson 5y ago
> Commercial software engineering is a creative endeavor; it is not a science, nor is it a manufacturing process. Do you think software engineering will remain a creative endeavour in the future? We've only been programming in high level languages for 60-70 years. All the while Moore's law has been in effect.
- spaetzleesser 5y ago
  “Do you think software engineering will remain a creative endeavour in the future?” I would think so. If something can be automated or be done with a reproducible process it will get automated or performed by lower paid people. But I can easily see how software and automated processes will reach a point of “good enough” for most purposes sometime in the future and there will be much less software engineers doing creative work. For example I would expect a lot of repetitive front end backend work will go away.

kragen 5y ago

"View Source" may be helpful if you're just seeing a blank page.

truenindb 5y ago

this problem where published science is often wrong and you don't need to actually be an engineer that history will not record the name of is completely new, and also, airplanes are impossible and where actually invented by The Wright Bros, because they had the Wright stuff, right?

shusson 5y ago

> ...“bugs found in requirements are 100x cheaper than bugs found in implementations.” ... There’s one tiny problem with the IBM Systems Sciences Institute study: it doesn’t exist It's funny to think how much of an effect this chart had on software engineering as a whole. I remember learning it at university and until today I thought it had some basis in science.

eschneider 5y ago
It "feels right" and is therefore "truthy."
- krisoft 5y ago
  Do you guys doubt that it is right? What kind of information would you require to show you that it is true? I'm one of the lucky ones who tends to work on safety critical "complex cyber-physical systems". Is that maybe the difference? There is the case where a developer runs unit tests before a commit. Some of the tests are unhappy. Developer fixes it and commits it up. Maybe hours spent? Then there is the case where something goes funky in the real-world all-up integrated system at the test range. Even in the best case tens of people waste a day. If we are unlucky it is a hard one and a small army of the most senior developers with the best operators and hardware people are hunting the bug for weeks. I fear to even sum op the wasted cumulative work days. I lived through multiple of these at different companies on different projects. (Both the first kind and the second kind.) Is the question if this is true? Or maybe the question is if this is true for your area of the industry too? Obviously I won't be able to provide sources for the ones I worked on. I bet no company really would want to release the hard data on these things. So let's look at some well-publicised problem caught in production: The Boeing 737 MAX MCAS issue. 346 deaths, 1 year, 8 months and 5 days grounding and the estimated direct costs are US$20 billion. Obviously this is all super anecdotal. I just want to understand what part of the question/problem you have doubts about.
  - twobitshifter 5y ago
    I think that’s the point - the chart confirm your bias and everyone else’s so we accept the results at face value. Nobody really knows if those bar heights were ever measured but of the millions of people to see that chart few questioned it.

qznc 5y ago

Greg Wilson is more positive. Though he does agree that nobody cares. https://third-bit.com/2021/07/17/software-engineerings-greatest-hits/ https://third-bit.com/2021/07/17/software-engineerings-great...

dang 5y ago
That (or more likely https://www.youtube.com/watch?v=HrVtA-ue-x0 https://www.youtube.com/watch?v=HrVtA-ue-x0 since that's where the info is) would make a good HN submission in its own right. Probably best to wait a few days to let the hivemind caches clear (followup/copycat posts aren't great: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=follow-up%20by%3Adang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). If you submit it, let us know at hn@ycombinator.com so we can put it in the second-chance pool (https://news.ycombinator.com/pool https://news.ycombinator.com/pool, explained at https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308), so it will get a random placement on HN's front page.
- hwayne 5y ago
  This is a cool system!
jdlshore 5y ago
I'm skeptical of Wilson's reporting. One of his quoted studies looked interesting (slide 20, Fucci 2016, "An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach"), so I looked it up. Wilson describes that study as involving 39 professionals on real projects. But when I looked at the actual study [1], it involved 21 graduate students working on two toy problems, the Bowling Game kata and "Mars Rover API." This is a disappointing misrepresentation of the study. A charitable interpretation is that he was referring to a different study, and accidentally put the wrong reference on that slide. Or maybe the actual talk explains the difference. I'm not sure—I didn't look deeper. [1] http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf http://people.brunel.ac.uk/~csstmms/FucciEtAl_ESEM2016.pdf

deleted 5y ago

[deleted]

blunte 5y ago

> But first you gotta find the primary sources. Yeah, and the real primary sources are the companies building and running software. They don't have time or interest in the long term research. And likely they aren't even concerned with their own technical situation 12 months in advance. They just need to "deliver", they think. So unless you can find some clever way to gather all their data (Sentry, AppSignal, ...?), you will never know what's really happening in most places. Your research will be artificial without that real world data.

lifeisstillgood 5y ago

My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. Software is more like a novel than a bridge. Maybe, maybe we can apply engineering terms to machine code or assembly. But the abstract levels we all work at - not a hope. I mean this is a good thing. It changes how we think about software - engineering has changed the world - and it's in the hands of a few professionals. Imagine software in the hands of ... everyone. It's like Eric Idle gather mud in the Dark Ages imagining what a literate world that reads the New york times will be like

hwayne 5y ago
> My two cents: Software is a form of literacy not engineering. One good bridge looks and acts almost exactly like every other bridge in the world. Good software that does a given job can be unrecognisable compared to other good software that does exactly the same job. 1. This is false. One of the reasons making bridges is so expensive is because each bridge is a precious special snowflake with lots of challenges specific to that bridge project and no other bridges. 2. Bridge building is a very small part of civil engineering, and civil engineering is a very small part of traditional engineering. Maybe software is more like process engineering, or chipset design, or subsystem integration, or workflow optimization.
- lifeisstillgood 5y ago
  1. I agree bridges have special challenges, but ... well for 2,000 years there was a keystone, and an arch. Yes the soil and the land were challenging, but I find it hard to argue that every stone bridge was a special snowflake. Snap a supporting cable on any suspension bridge and they all have the same failure mode. I think we are stretching analogies - is "build a CRM for the sales department" simple as in "there is a keystone data design common across all CRMS - and all the rest is "shifting sands of the river are same as shifting alliances in the boardroom" I just try to keep this simple. Coding uses symbolic representation to describe a model - this is what language is. Compilers do / can do things that simply don't exist conceptually in Engineering. Coder manipulate symbols. It really is a language thing. 2. yes. I was just using bridges as an example of engineering. As CRUD websites are an example of software engineering.

Quanttek 5y ago

> The average developer thinks empirical software engineering is a waste of time. How can you possibly study something as complex as software engineering?! You’ve got different languages and projects and teams and experience levels and problem domains and constraints and timelines and everything else. Why should they believe your giant incoherent mess of sadness over their personal experience or their favorite speaker’s logical arguments? This argument could be used for literally any science but, for some reason, it seems to fall on attentive ears mostly in the software industry.

hwayne 5y ago
I think it's common in a lot of fields. It's def prevalent in medicine, for example: people are skeptical of any research on doctor behavior.
twobitshifter 5y ago
I think the issue is that management is not done scientifically. You can see this with something as estimating for a project. Higher ups just ask “how many hours and when can you get it done” - they don’t want you to go through past similar work review it’s timeliness and hours, divide it into smaller tasks, build a project schedule, review productivity trends and resource loadings, or any scientific way of arriving at the number. What they really want is a seat of the pants WAG, which isn’t too big to scare the client, that they can use to get going, with the understanding that shit happens along the way.

cycomanic 5y ago

I found the complains about how reading papers, do literature research, evaluating papers and the field etc., is difficult somewhat amusing. That's actually what a large portion of the PhD education and becoming an academic is about. Yes, it is not easy, if it was a PhD wouldn't take between 3 and 6 years. Generally, the author makes some good points and there is definitely often a disconnect between academia and industry. I think it is important to remember that science is much more like a directed random walk toward an unknown goal. If industry wants specific answers to specific questions they could (should) finance the studies that provide them the answers. However, my impression is that when industry finances significant studies it is much more toward validating/confirming their already established practices, products... (the topic of these "bought" studies is another can of worms in the topics of academia and industry interactions)

mcguire 5y ago

"Here’s the only technique I’ve found that works, which I call scrobbling even though that means something totally different: "1. Search seed terms you know, like “cost of bugs”, in an appropriate journal (here’s one). "2. Find papers that look kinda relevant, skim their abstracts and conclusions. "3. Make a list of all the papers that either cite or are cited by these papers and repeat. "4. Find more useful terms and repeat." Congratulations! You're a grad student now! P.S. When I graduated, I had a filing cabinet with two drawers full of papers that felt important enough to save printed copies, from just this process. A couple of months ago, I carted them out to the burn pile, threw them on the pile of leaves and brush, and held a Viking funeral. And I didn't even feel bad. :-)

hwayne 5y ago
I got into software specifically because I didn't want to go to physics grad school, and now I read more papers than I ever did in physics. Man plans, God laughs.
titzer 5y ago
My impression on reading this list was, uh yeah, there's no avoiding that. Except if you know an expert. Then just ask an expert. Search engines for papers suck, bad. At the fringes of human knowledge, you need an actual brain to process cutting-edge research. Either your own non-expert brain, or an expert brain to help guide you. That process is the same in every field. Older fields have textbooks with knowledge distilled from experts. That's a good place to start to get the state of the art > 20 years ago.
- frickinLasers 5y ago
  | Search engines for papers suck, bad. This screams the need for an intelligent search engine, with summarizing features that point out relevant parts of the paper, and (imho) a UI and algorithms that reach across disciplines and encourage discovery. So, for instance (I'm making this up), someone looking for a way to understand interferometry data might stumble on a useful regularization technique from image processing. I imagine academic libraries would pay really good money for that. Of course, it would help if a mostly complete body of scientific literature was available to crawl--like SciHub.

TeeMassive 5y ago

The only truths that I've known so far to be true: 1) Small patches with quick feedback 2) Automate the manual stuff, even if you're not doing it often you'll probably like having that script ready and tested when you'll have to do it again in one year.

walshemj 5y ago

The Bug Vs Defect made me laugh back at my first job we partnered with the BSI to do research that went into BS5750 / ISO 9000. The Head of software side at the BSI commented that tat for old timers in QA - software was odd as they "did not know wear to hang the defect tags"

aranchelk 5y ago

> The average developer thinks empirical software engineering is a waste of time. Bold claim. Any data at all to back that up? My guess based on experience would be most developers are unfamiliar with the term and haven’t made a judgement one way or the other.

aeturnum 5y ago

Software engineering research is immature and, as such, has few (no?) strongly-supported claims that the entire field relies on. I agree this is a frustrating state of the field, but the way this is written makes the author appear to be unfamiliar with the state of things. For example: >Did I mention that all three of those papers use different definitions of “defect”? This kind of thing is really common in academia. I do not know software engineering research in detail, but there are older and more basic definitional fights in the social sciences. Defining a defect, as I'm sure the author of the blog would agree, is not a trivial decision. In general, this piece contains (perhaps hyperbolic?) dismissals of people trying to do work. It doesn't seem to respect what I imagine is earnest effort. Taken all together, it seems like the author either thinks the entire field is charlatans (which I doubt as they use the fields' conclusions at the end) or is making light of people working on problems they themselves do not have answers to in order to have a more lively writing style. It was fun to read and left a bad taste in my mouth.

BadInformatics 5y ago
As mentioned in TFA, the author has both written and spoken about software engineering research and related topics in some depth before. Though fiery, I think it does its job as a more informal cautionary tale on accepting research in the field without a critical gaze. Analyses like [1] and [2] (from the same author) are extraordinarily valuable because they help us wade through the firehose of published studies to find what (if any) conclusions may be gleaned. Without such a filter, we are fated to a) repeat work by slogging through the torrent ourselves, or b) taking stuff at face value. Certainly there is no shortage of the latter happening on forums like this, and I wonder how many misinformed decisions are made as a result. [1] https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ [2] https://www.hillelwayne.com/post/are-we-really-engineers/ https://www.hillelwayne.com/post/are-we-really-engineers/

Joeri 5y ago

I went into this software engineering productivity research rabbit hole a decade ago and came away similarly depressed with the lack of rigor in our field. What I’ve come to believe: - Software engineering is not an engineering discipline, it is a craft. It may or may not one day become an engineering discipline, I find this impossible to tell. - The difference between a feature and a bug is largely one of semantics, not of substance. A defect is a mismatch between expected behavior and actual behavior. A bug is when the expectation was that of the programmer that wrote the code. A feature request is when the expectation was that of a user. There is no real difference for the user between a missing feature and a broken feature. Hence why we so often get side-tracked into fruitless “bug or feature” discussions and why users don’t care when we say it is by design. - Errors will happen. People are flawed and produce errors at a more or less set average rate for each person (unless they’re tired or enjoying a ballmer peak). Errors in every step of the process, from design to deployment, are inevitable. We can however catch these inevitable errors, with automated error detection tools, or by having a fresh set of eyes look at the work. The consequence is that making high quality software necessitates taking a look at the whole process and introducing error detection steps in every part of it, whether they be automated like linters and unit tests, or human like design and code review or mentoring and pair programming. - Errors in design are more costly because they cause entire features to need to be written or rewritten, which is the most costly kind of defect to resolve. We do not need research to know this to be true. We solve this by having short feedback loops with real users, which is the only real form of agile development.

goodpoint 5y ago
> Software engineering is not an engineering discipline, it is a craft Software engineering is *still not treated* as an engineering discipline, it is *still treated as* a craft

john579 5y ago

Bugs in software are difficult to qualify, both in detection and resolution. Find a bug, fix it. Can't fix it? Change the specs or call it a feature. Otherwise call the program obsolete and rewrite it. For example, a new technology such as GPGPU comes out. Now programs can run much faster. This changes the problem, the approach, the implementation, the design, the requirements, and the solution. Perhaps nothing was wrong with the original CPU code, but the GPU introduced slowness bugs into the CPU code. What's the cost of fixing that bug? The cost of redesigning the problem, rewriting the code, and writing a GPU debugger so you can fix those new bugs. New technology brings change. Your faster solution brought about social change, which caused rioting, then nuclear holocaust, then the power goes out, and the whole process repeats. Add up the value of the entire world, civilization, and history, and that's the cost of fixing that bug. That's software engineering research.

bob1029 5y ago

Software engineering can only exist with an incredible amount of discipline. We routinely start new implementations in excel and perform formal normalization checks over models before writing any code. You can get away without having a strong type system as long as your tables and relations are clean. This stuff is so powerful. Figuring out how to funnel your domain instances into a SQL db and writing all your complex logic in terms of basic queries is how you can build an ecosystem in which determinism and formal proofs of correctness are possible. For me, software engineering is about correctness for the business end. There is certainly still a layer of craftsmanship below the engineering realm which enables all the magic. This is where things like performance and reliability live. Interacting with the outside world is an icky thing, and the specific implementation always depends on the use case.

feoren 5y ago
Software engineering is so weird -- what you've just described sounds awful to me. I believe almost the exact opposite of everything you've said. Yet I have no doubt* you've seen good results from your approach, and I get good results from mine. It's surprising how personal software engineering really is. * I do actually have doubts, because 99.9% of software is utter garbage. But I'm trying to be more positive and the point stands whether you're in the 99.9% or the 0.1%.
- bob1029 5y ago
  Willingness to tolerate formal methods does not mean you always have to endure pain and suffering. On the contrary, you may find that you are shipping software that is so good you begin to see the value. For a lot of shops, the problem domain probably isn't complex enough to justify intense formality in the process (i.e. engineering). Once you are dealing with monsters like supply chain or factory automation, you can't play games anymore.
  - Epa095 5y ago
    > On the contrary, you may find that you are shipping software that is so good you begin to see the value. You may also find that when you finally do ship, everyone had started using your competitor, or that your customers needed something slightly different, or that the cost of waiting for you was so high it would have been better having something early. I have great respect for formal verified software, I have even touched on it a bit during my PhD, and there are definitely places for where it makes sense. Same for formal methods in general. But there are other software tasks where it is not just 'not needed' but it is actually the wrong choice. It depends a lot on the opportunity cost of delivering later, and on how well defined the tasks are, and how preffesional the customer is. And the tasks where less formal methods are the right way are not "playing games", they are solving the problem under a different set of constrains, also doing software engineering.

lincpa 5y ago

The Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model Follower's success stories: Apple M1, HPE Cray SmartSim Come soon: Qualcomm, MS, Intel, AMD, Samsung, Cray, etc. https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

tdeck 5y ago

Somewhat related: I enjoyed reading the eBook "The Leprechauns of Software Engineering" by Laurent Bossavit [1]. The author tried to track down the source research behind various pieces of received wisdom. I remember that the claimed cost of fixing bugs at a later stage was one of them. [1]: https://leanpub.com/leprechauns https://leanpub.com/leprechauns

BerislavLopac 5y ago

https://news.ycombinator.com/item?id=27891102 https://news.ycombinator.com/item?id=27891102