14 ms·

Evidence-based software engineering: book released

290 points by qznc 6y ago

spekcular 6y ago
I read the entirety of a draft of this once, about a year ago, when I was bored one night. I don't want to be overly critical of someone's book, so let me just say that: A) I don't understand why an entire elementary statistics pseudo-textbook is bolted on at the end, forming the entire back half of the text, and B) the title interested me because it promised concrete information that would improve my own software development, and while I found many things in the first half of the book, I didn't find this.
- pmiller2 6y ago
  Can you expand a bit on point B? What sort of things did you find in this book, and why were you not able to apply them to your own practice of software engineering?
  - spekcular 6y ago
    Sure. Chapter 1 is titled "Human Cognition" and reads like notes to an introductory psychology textbook. This is not a bad thing (psychology is fun to read about), but I didn't find any practical guidance about building software. Chapter 2 is "Cognitive Capitalism." Here, I find a lot of social and economic commentary, but again, no information that will actually assist me in writing software. The same story plays out for basically every other chapter, until we hit the material about statistics in the second half. Another way to say it is that the level of abstraction feels a bit off, resulting in the sense that the discussion is somewhat superficial and disconnected from practice. It's like someone wrote a book on gardening that begins with a 200 page discussion of cell biology.
- vector_spaces 6y ago
  The "statistics textbook" part is especially confusing to me -- who is the audience for this, and why is it even here? He mentions at the beginning of the probability chapter: "Readers are assumed to have some basic notion of the concepts associated with probabilities, and to have encountered the idea of probability in the form of likelihood of an event occurring" This doesn't give us any insight into what background is expected here -- do I need to have taken a measure theory based probability course? Or would a calculus-based course suffice? What about a course for non-STEM majors? The word "basic" means different things to different people, and is generally an anti-pattern in mathematical writing. It's meaningless at best, and gatekeeping at worst. Then he launches into this downright strange rabbit hole about mental models. None of this discussion in this chapter feels properly motivated, the writing is disjointed, and it reads like an unedited, non-proofread stream of consciousness. In particular, the exposition gets too distracted by its own examples. Don't get me wrong -- many of the paragraphs in isolation are interesting, but taken together it's excruciating to read. It's an impressive project overall, and certainly ambitious, but it falls down due to the writer's lack of empathy for the reader. I suspect the situation would be better if they reduced the scope of the project and kept the writing more focused.
- asphaltycode 6y ago
  > concrete information that would improve my own software development That’s what I was looking for when I clicked the link here. Do you have any recommendations for that kind of stuff, evidence-based if possible?
  - gmargari 6y ago
    I would suggest the Clean Code/Coder/Architecture series of Robert Martin.
- rainforest 6y ago
  > I don't understand why an entire elementary statistics pseudo-textbook is bolted on at the end, forming the entire back half of the text It's quite difficult to talk about empirical software engineering without discussing methods, after all papers like [1] were deemed necessary 20 years ago and still the occasional meta-paper is published about correct design of experiments or analyses. As someone who worked in the field it doesn't seem particularly surprising to see some treatment - there are a handful of papers in my former subfield that are oft-cited because they describe a statistical procedure/experiment design consideration, but they also bundle the explanatory stats "for free". I would hazard a guess that the intent of these chapters is to equip the reader with enough background that they could replicate or run some of the experiments in the book to try to specify findings/experiments to their own organisations. I'd follow that with an assumption that the author felt that chapter 13 needed background, and recursed until they'd finished writing a textbook. [1] Kitchenham et al. "Preliminary guidelines for empirical research in software engineering" 2001: http://www.ehealthinformation.ca/wp-content/uploads/2014/07/2001-Preliminary-Guidelines-for-Empirical-Research.pdf http://www.ehealthinformation.ca/wp-content/uploads/2014/07/...
anonuser123456 6y ago
Anytime I see 'evidence based', I can't help but think "my opinion for which I've cherry picked data".
- Shugarl 6y ago
  I also find it quite strange. I remember my teachers telling that we're making a little bit of progress in software engineering research, but comparing it to the state of research in fields like Math or Physics would be like comparing a fully grown man and a toddler. In other words, we hardly know anything
  - pmiller2 6y ago
    We may hardly know anything, but, comparing any field that involves humans to math or physics is an apples to oranges comparison. All it takes to do math research is time, knowledge, writing material, and, preferably, the support of an institution which allows one to do research full time. There are no pesky experiments to run, except possibly inside a computer. Physics can be the same way, but, even when it involves "pesky experiments," those experiments only depend on the physical laws of the universe, and the labor of a fleet of graduate students who have great incentives to see these experiments through. SWE research has either of these properties.
- st1x7 6y ago
  What's the alternative?
  - dpc_pw 6y ago
    Stop pretending that there's an easy empirical answer to everything, and accept the nature of complex domains.
- mooreds 6y ago
  Have you checked out the book (PDF here: http://knosof.co.uk/ESEUR/ESEUR.pdf http://knosof.co.uk/ESEUR/ESEUR.pdf )? There are 2000 footnotes referencing published studies. It's not perfect, but hopefully there are enough different views all triangulating on "truth" to provide value.
chubot 6y ago
Previous thread: https://news.ycombinator.com/item?id=23769930 https://news.ycombinator.com/item?id=23769930
splittingTimes 6y ago
So how do you measure performance or success of an engineering department within a bigger corp? Not to evaluate salaries or budgets, but to know, are we moving in the right direction? Are we improving or are we getting worse? From what I see most metrics on an individual or team level can/will be gamed leading to worse outcomes. Metrics on higher levels like on the level of the product or even revenue of the company/product depends on many other players outside of the control of the engineering team (product management, operations, QA, regulatory, marketing, sales, etc). This of course it's due to the fact that delivering value to customers is a team effort. But how can you know/show your department is on the right track?
- save_ferris 6y ago
  Personally, having worked for a few big companies in my career, I believe it's absolutely possible for a company to be too big. It might be an impious opinion, but there absolutely comes a time in a company's existence that internal politics begins to undermine the mission of the entire organization. Of course, I haven't worked everywhere and I recognize that this opinion is based purely on my own experience. But companies that generate billions upon billions in revenue seem to care less and less about functional process improvement and more about short-term profit gains. Because ultimately, the C suite is paid based on stock performance for most large companies, and that becomes the only metric that matters. In that situation, everything falls in line behind it.
- solidasparagus 6y ago
  Why do you want to measure the productivity of the engineers in isolation when their ability to deliver value depends on other people? Too often the answer to that questions boils down to 'ammunition for the blame game'. If your engineers can't deliver value because of say product, who cares how productive they are? Measure business outcomes - you will get what you measure. If engineering is being measured by the ability to deliver business outcomes but engineers can't because of other people, the software engineers are going to let you know* and it's an important thing to learn about and fix. *this assumes you have competent managers and engineers who are willing to communicate with those managers, but if you don't have that, this is sort of a moot point.
aard 6y ago
> "In a competitive market for development work and staff, paying people to learn from mistakes that have already been made by many others is an unaffordable luxury; an engineering approach, derived from evidence, is a lot more cost-effective than craft development." I disagree with this characterization. Yet again, we developers are being told to just follow the program. That programming is not an artistic or craft endeavor that benefits from experience and intuition. It is like working in a factory where coders just bang out widgets on an assembly line. And the self-appointed thinkers will optimize the process for us. What is at risk by not allowing developers to "learn from mistakes" is autonomy. Striping developers of their autonomy is the primary cause of poor performance, not an inability to execute so-called "best practices" Attempts to codify the process of software development always fail, because coding is a design process not a manufacturing one. Developers do their jobs in many different ways, many of which are equally effective. There is more than one way to skin a cat -- especially in creative work. > "The labour of the cognitariate is the means of production of software systems" This false assumption is at the base of the problem. The work of the compiler (or scp) are the means of software production. Coding is design. Once the design is complete, the results are compiled and copied to their target environments. In software, production is negligible. Which promotes the misconception that developers are producing software. In actuality they are designing software. The difference may seem subtle, but it is crucial.
- Jtsummers 6y ago
  It really depends on the kind of systems and work you're doing. In my previous office, there was an entire group (about 100-150) of programmers whose job was incredibly rote. That is, you could take a novice out of college and get them up to speed in about 1-3 months to do even their most complex work. However, the other groups were much less factory-ish, though rarely anything truly novel. Only a small cadre of programmers were working on anything that really required novelty and creativity. It's a spectrum and that has to be understood by all participants in the discussion. Managers want everything to be like that first group, because it's so consistent and predictable. They want to know that a project will take 1000 man-hours and be right 95% of the time. Many programmers want to see themselves part of the last group. The reality is, most of us are in the middle. There are aspects of the job that are almost mechanical, and aspects which require greater creativity or "craft". If we can clarify that for the managers, it can go a long way to ending some of the nonsense.
jorgBaller 6y ago
I find the use of evidence concerning. Software engineering best practices should be established by highly educated academics (Phds etc).
- AnimalMuppet 6y ago
  What are you trying to say? I honestly can't tell. It sounds like you're saying that PhDs should decide what best practices are, without regard for evidence. If so, that had better be sarcasm, because it's patently absurd. If your point was something else, could you clarify?
  - Jtsummers 6y ago
    I assumed they were being sarcastic. Perhaps a reference to groups like the SEI (Software Engineering Institute).
    - jorgBaller 6y ago
      That was subtle for most people to pick up on. Congrats!
eatonphil 6y ago
Is there a physical copy I can buy (couldn't find one) or do I have to build one from Lulu?
- Splendor 6y ago
  > I’m investigating the possibility of a printed version. That's the 3rd sentence in the linked page.
mettamage 6y ago
Looking at the TOC this doesn't feel like evidence-based software engineering. This is simply because there seems to be way too much background in this book. It basically spans 4 topics: - Cognitive (neuro)psychology - Economics / finance - chapter 4 to 7 seem to be about software engineering - Statistics Don't get me wrong, I'll take a read, but not because I want to read about evidence-based software engineering. It's much more because it seems that you're connecting a lot of different areas together and I want to see how you do it. The second reason is because I happen to know a lot about these specific areas as well :)
StillBored 6y ago
A bigger question is whether anyone would pay attention if it turns out some method has a notable improvement. I remember reading a bunch of code quality studies a few years back. A bunch of them were DoD studies from the 1980/1990's, and the one thing that stuck out were a couple of studies that looked at style issues with ada/c and noted that the matched brace style significantly lowered a couple different defect types. Yet today, particularly in the opensource world matched brace style is almost completely non-existant, since its a religious war thing, and some of the early opensource projects were run by people who didn't like it.
- bigbubba 6y ago
  I love matching braces, if for no other reason than because they make transversing code easier. If you ever find yourself at the bottom of an awful function with multiple screenfuls of code, matching braces makes it trivial to jump to the top if that function even when your editor has no special knowledge of that language. No LSP/etc needed. Barebones vanilla Vi is actually a viable editor for Lisp because of this, relative to Vi with languages like Python.
  - ywei3410 6y ago
    This is one of the reasons that many Lisp developers like Lisp code - local structured editing is trivial because if the braces match, then most of the parser doesn't need to deal with the rest of the file.
  - throwaway_pdp09 6y ago
    > of an awful function with multiple screenfuls of code If you've got that, you will have deeper and more abundant problems to solve than will be helped by easily matching up braces.
    - deckard1 6y ago
      heh. That reminds me of JavaScript recently. We have optional chaining now, which quite a few developers absolutely love. Diving deep into data structures is such a code smell, and yet so many people just want to spray the equivalent of Febreze over it. "Maybe with the right syntax for doing this ugly thing people won't notice how ugly this thing I'm doing actually is!"
tomgp 6y ago
If you’re looking for something a little more focused I can heartily recommend Andy Oram and Greg Wilson’s “ Making Software: What Really Works, and Why We Believe It” it’s a great collection of literature reviews around a lots of common software engineering practices.
beprogrammed 6y ago
Wow, what a wonderful pdf.
louiechristie 6y ago
Where’s the ‘conclusions’ section?
RoutinePlayer 6y ago
Evidence-base X is the best X humans can muster.. except for Love
p0nce 6y ago
Honestly sounds very interesting and necessary book.
Geminidog 6y ago
Have you guys ever tried software engineering engineering techniques that utilize No evidence? The current trend of the programming world is to worship data and science but some of the most powerful programming techniques do not involve data or science. A book taking another approach to programming: http://www4.di.uminho.pt/~jno/ps/pdbc.pdf http://www4.di.uminho.pt/~jno/ps/pdbc.pdf Some of you may have trouble distinguishing what is not evidence based and what is evidence based. Hopefully the following dichotomy can clear up any misunderstandings: Science is evidence based, Math is not evidence based. In the real world the ideal approach is to utilize both evidence and non-evidence based techniques, but more often then not the non-evidence based techniques are harder to understand and therefore under utilized.
StonyRhetoric 6y ago
I'm having a hard time deciding if this is the work of a well-meaning but misguided obsessive, or GPT-3 generated pseudo-gibberish. In any case, this book could use a great deal of editing. If there is wisdom in here, I don't want to dig through hundreds of pages to find it.
- d23 6y ago
  I’m glad I’m not the only one. I started trying to skim it and couldn’t figure out what the hell was going on. Granted, I didn’t give it a strong chance, but the title made it seem like something very different than what it is. I think.
jeff-davis 6y ago
Until we start having a "lines of code" budget, nothing will be fixed. If you want feature X, how many lines of code will it take? 2000? Oh, sorry, we are already spending 6000 lines of code on other features, so find a way to do it in 500 lines or defer the project. Yes, just like anything, this can be gamed. But at least it would get product managers thinking about the complexity they are imposing and the effect that will have on other features.
- UK-Al05 6y ago
  Sounds like way to form incredibly dense codebases.
arminiusreturns 6y ago
What I appreciate about this is how it addresses the more managerial aspects, things I've been trying hard to wrap my head around so that I can better approach management with things like data based risk analysis based on existing risk analysis algorithms, and things like how to evaluate certain interal functions efficacy. I will very much be reading this thoroughly. Now what I really need is a book on how to break through the ceiling to start communicating with the C's/board better (or at all in some companies where managers and C's are blackholes of information...) This is one of my largest career failures I constantly think about... I see failures that are managerial first but have second tier effects of causing technical issues (technical debt being only one), but everyone in the chain fails to know how to voice this to higher ups, who then tend to assume it's all technical or team based and do things like start replacing managers with PHB's. Sometimes I wish I had gotten an MBA instead just so I could navigate the culuture/speak better. How can an engineer play at the c/board level if he has the desire and aptitude? Any reading recommendations would be appreciated. The older I get the more I regret not "playing politics". Is there a "The Prince - but for SV?"
- DoingIsLearning 6y ago
  > how to break through the ceiling to start communicating with the C's/board better Anecdata and perhaps unfair but from my own observations, once you get to C-level, who is doing the communicating is much more important than how the message is conveyed. There is definitely a 'club' mentality, usually the way you will get a voice is by building relations with C-levels that will 'adopt' you as their protégé/mentee. Once you have someone in the room who can say "we should listen to this person" then you will have a chance of actually getting info/influence upstream. But to be build those relations there is a fair degree of 'self-promoting' and being seen as the go-getter in the crowd. Silently doing your work and silently going above and beyond is usually just a recipe for disappointment. Another thing I don't often see discussed is that, as with a lot of other things, luck and timing matters a hell of a lot.
  - beardedwizard 6y ago
    accurate. People struggle to navigate this realm without being disingenuous though. Stay true, but play the relationship game instead of the write code game.
deleted 6y ago
[deleted]
tdrgabi 6y ago
Hi Derek, thank you for writing the book and releasing it. I haven't seen a similar book before for software engineering, so I'm very intrigued and intend to read it. I feel bad for asking something from a free resource, that took a long time to make and put together, so please ignore it if it's a lot of trouble. Is it possible to publish epub or mobi formats of the book? That would help ereaders format it better, pdf's are inflexible. If it's not easy, please ignore this request. Thank you for writing the book.
osullivj 6y ago
"The craft approach has survived because building software systems has been a sellers market, customers have paid what it takes because the potential benefits have been so much greater than the costs." Gotta disagree here. SW dev has never been a seller's market in the way that eg the property market or job market can be a seller's or buyer's market. There are powerful effects in the market making that a massive simplification. Firstly platform network effects; think of the browser wars for instance. Secondly, open source: a lot can be had for "free". Thirdly, lowest bidder: John Glenn famously commented, when asked what he thought about sat on top of an Apollo rocket, "every component has been built by the lowest bidder". Fourthly, novelty: a significant amount of SW dev is about building things that haven't been built before. Which can be more like a research project than eg putting up a new apartment block, or bridge, or designing and productionising a new car.
woile 6y ago
Is it available in a kindle format? mobi, or epub would be amazing!

290 points by qznc 6y ago

I read the entirety of a draft of this once, about a year ago, when I was bored one night. I don't want to be overly critical of someone's book, so let me just say that: A) I don't understand why an entire elementary statistics pseudo-textbook is bolted on at the end, forming the entire back half of the text, and B) the title interested me because it promised concrete information that would improve my own software development, and while I found many things in the first half of the book, I didn't find this.

pmiller2 6y ago
Can you expand a bit on point B? What sort of things did you find in this book, and why were you not able to apply them to your own practice of software engineering?
- spekcular 6y ago
  Sure. Chapter 1 is titled "Human Cognition" and reads like notes to an introductory psychology textbook. This is not a bad thing (psychology is fun to read about), but I didn't find any practical guidance about building software. Chapter 2 is "Cognitive Capitalism." Here, I find a lot of social and economic commentary, but again, no information that will actually assist me in writing software. The same story plays out for basically every other chapter, until we hit the material about statistics in the second half. Another way to say it is that the level of abstraction feels a bit off, resulting in the sense that the discussion is somewhat superficial and disconnected from practice. It's like someone wrote a book on gardening that begins with a 200 page discussion of cell biology.
vector_spaces 6y ago
The "statistics textbook" part is especially confusing to me -- who is the audience for this, and why is it even here? He mentions at the beginning of the probability chapter: "Readers are assumed to have some basic notion of the concepts associated with probabilities, and to have encountered the idea of probability in the form of likelihood of an event occurring" This doesn't give us any insight into what background is expected here -- do I need to have taken a measure theory based probability course? Or would a calculus-based course suffice? What about a course for non-STEM majors? The word "basic" means different things to different people, and is generally an anti-pattern in mathematical writing. It's meaningless at best, and gatekeeping at worst. Then he launches into this downright strange rabbit hole about mental models. None of this discussion in this chapter feels properly motivated, the writing is disjointed, and it reads like an unedited, non-proofread stream of consciousness. In particular, the exposition gets too distracted by its own examples. Don't get me wrong -- many of the paragraphs in isolation are interesting, but taken together it's excruciating to read. It's an impressive project overall, and certainly ambitious, but it falls down due to the writer's lack of empathy for the reader. I suspect the situation would be better if they reduced the scope of the project and kept the writing more focused.
asphaltycode 6y ago
> concrete information that would improve my own software development That’s what I was looking for when I clicked the link here. Do you have any recommendations for that kind of stuff, evidence-based if possible?
- gmargari 6y ago
  I would suggest the Clean Code/Coder/Architecture series of Robert Martin.
rainforest 6y ago
> I don't understand why an entire elementary statistics pseudo-textbook is bolted on at the end, forming the entire back half of the text It's quite difficult to talk about empirical software engineering without discussing methods, after all papers like [1] were deemed necessary 20 years ago and still the occasional meta-paper is published about correct design of experiments or analyses. As someone who worked in the field it doesn't seem particularly surprising to see some treatment - there are a handful of papers in my former subfield that are oft-cited because they describe a statistical procedure/experiment design consideration, but they also bundle the explanatory stats "for free". I would hazard a guess that the intent of these chapters is to equip the reader with enough background that they could replicate or run some of the experiments in the book to try to specify findings/experiments to their own organisations. I'd follow that with an assumption that the author felt that chapter 13 needed background, and recursed until they'd finished writing a textbook. [1] Kitchenham et al. "Preliminary guidelines for empirical research in software engineering" 2001: http://www.ehealthinformation.ca/wp-content/uploads/2014/07/2001-Preliminary-Guidelines-for-Empirical-Research.pdf http://www.ehealthinformation.ca/wp-content/uploads/2014/07/...

anonuser123456 6y ago

Anytime I see 'evidence based', I can't help but think "my opinion for which I've cherry picked data".

Shugarl 6y ago
I also find it quite strange. I remember my teachers telling that we're making a little bit of progress in software engineering research, but comparing it to the state of research in fields like Math or Physics would be like comparing a fully grown man and a toddler. In other words, we hardly know anything
- pmiller2 6y ago
  We may hardly know anything, but, comparing any field that involves humans to math or physics is an apples to oranges comparison. All it takes to do math research is time, knowledge, writing material, and, preferably, the support of an institution which allows one to do research full time. There are no pesky experiments to run, except possibly inside a computer. Physics can be the same way, but, even when it involves "pesky experiments," those experiments only depend on the physical laws of the universe, and the labor of a fleet of graduate students who have great incentives to see these experiments through. SWE research has either of these properties.
st1x7 6y ago
What's the alternative?
- dpc_pw 6y ago
  Stop pretending that there's an easy empirical answer to everything, and accept the nature of complex domains.
mooreds 6y ago
Have you checked out the book (PDF here: http://knosof.co.uk/ESEUR/ESEUR.pdf http://knosof.co.uk/ESEUR/ESEUR.pdf )? There are 2000 footnotes referencing published studies. It's not perfect, but hopefully there are enough different views all triangulating on "truth" to provide value.

chubot 6y ago

Previous thread: https://news.ycombinator.com/item?id=23769930 https://news.ycombinator.com/item?id=23769930

splittingTimes 6y ago

So how do you measure performance or success of an engineering department within a bigger corp? Not to evaluate salaries or budgets, but to know, are we moving in the right direction? Are we improving or are we getting worse? From what I see most metrics on an individual or team level can/will be gamed leading to worse outcomes. Metrics on higher levels like on the level of the product or even revenue of the company/product depends on many other players outside of the control of the engineering team (product management, operations, QA, regulatory, marketing, sales, etc). This of course it's due to the fact that delivering value to customers is a team effort. But how can you know/show your department is on the right track?

save_ferris 6y ago
Personally, having worked for a few big companies in my career, I believe it's absolutely possible for a company to be too big. It might be an impious opinion, but there absolutely comes a time in a company's existence that internal politics begins to undermine the mission of the entire organization. Of course, I haven't worked everywhere and I recognize that this opinion is based purely on my own experience. But companies that generate billions upon billions in revenue seem to care less and less about functional process improvement and more about short-term profit gains. Because ultimately, the C suite is paid based on stock performance for most large companies, and that becomes the only metric that matters. In that situation, everything falls in line behind it.
solidasparagus 6y ago
Why do you want to measure the productivity of the engineers in isolation when their ability to deliver value depends on other people? Too often the answer to that questions boils down to 'ammunition for the blame game'. If your engineers can't deliver value because of say product, who cares how productive they are? Measure business outcomes - you will get what you measure. If engineering is being measured by the ability to deliver business outcomes but engineers can't because of other people, the software engineers are going to let you know* and it's an important thing to learn about and fix. *this assumes you have competent managers and engineers who are willing to communicate with those managers, but if you don't have that, this is sort of a moot point.

aard 6y ago

> "In a competitive market for development work and staff, paying people to learn from mistakes that have already been made by many others is an unaffordable luxury; an engineering approach, derived from evidence, is a lot more cost-effective than craft development." I disagree with this characterization. Yet again, we developers are being told to just follow the program. That programming is not an artistic or craft endeavor that benefits from experience and intuition. It is like working in a factory where coders just bang out widgets on an assembly line. And the self-appointed thinkers will optimize the process for us. What is at risk by not allowing developers to "learn from mistakes" is autonomy. Striping developers of their autonomy is the primary cause of poor performance, not an inability to execute so-called "best practices" Attempts to codify the process of software development always fail, because coding is a design process not a manufacturing one. Developers do their jobs in many different ways, many of which are equally effective. There is more than one way to skin a cat -- especially in creative work. > "The labour of the cognitariate is the means of production of software systems" This false assumption is at the base of the problem. The work of the compiler (or scp) are the means of software production. Coding is design. Once the design is complete, the results are compiled and copied to their target environments. In software, production is negligible. Which promotes the misconception that developers are producing software. In actuality they are designing software. The difference may seem subtle, but it is crucial.

Jtsummers 6y ago
It really depends on the kind of systems and work you're doing. In my previous office, there was an entire group (about 100-150) of programmers whose job was incredibly rote. That is, you could take a novice out of college and get them up to speed in about 1-3 months to do even their most complex work. However, the other groups were much less factory-ish, though rarely anything truly novel. Only a small cadre of programmers were working on anything that really required novelty and creativity. It's a spectrum and that has to be understood by all participants in the discussion. Managers want everything to be like that first group, because it's so consistent and predictable. They want to know that a project will take 1000 man-hours and be right 95% of the time. Many programmers want to see themselves part of the last group. The reality is, most of us are in the middle. There are aspects of the job that are almost mechanical, and aspects which require greater creativity or "craft". If we can clarify that for the managers, it can go a long way to ending some of the nonsense.

jorgBaller 6y ago

I find the use of evidence concerning. Software engineering best practices should be established by highly educated academics (Phds etc).

AnimalMuppet 6y ago
What are you trying to say? I honestly can't tell. It sounds like you're saying that PhDs should decide what best practices are, without regard for evidence. If so, that had better be sarcasm, because it's patently absurd. If your point was something else, could you clarify?
- Jtsummers 6y ago
  I assumed they were being sarcastic. Perhaps a reference to groups like the SEI (Software Engineering Institute).
  - jorgBaller 6y ago
    That was subtle for most people to pick up on. Congrats!

eatonphil 6y ago

Is there a physical copy I can buy (couldn't find one) or do I have to build one from Lulu?

Splendor 6y ago
> I’m investigating the possibility of a printed version. That's the 3rd sentence in the linked page.

mettamage 6y ago

Looking at the TOC this doesn't feel like evidence-based software engineering. This is simply because there seems to be way too much background in this book. It basically spans 4 topics: - Cognitive (neuro)psychology - Economics / finance - chapter 4 to 7 seem to be about software engineering - Statistics Don't get me wrong, I'll take a read, but not because I want to read about evidence-based software engineering. It's much more because it seems that you're connecting a lot of different areas together and I want to see how you do it. The second reason is because I happen to know a lot about these specific areas as well :)

StillBored 6y ago

A bigger question is whether anyone would pay attention if it turns out some method has a notable improvement. I remember reading a bunch of code quality studies a few years back. A bunch of them were DoD studies from the 1980/1990's, and the one thing that stuck out were a couple of studies that looked at style issues with ada/c and noted that the matched brace style significantly lowered a couple different defect types. Yet today, particularly in the opensource world matched brace style is almost completely non-existant, since its a religious war thing, and some of the early opensource projects were run by people who didn't like it.

bigbubba 6y ago
I love matching braces, if for no other reason than because they make transversing code easier. If you ever find yourself at the bottom of an awful function with multiple screenfuls of code, matching braces makes it trivial to jump to the top if that function even when your editor has no special knowledge of that language. No LSP/etc needed. Barebones vanilla Vi is actually a viable editor for Lisp because of this, relative to Vi with languages like Python.
- ywei3410 6y ago
  This is one of the reasons that many Lisp developers like Lisp code - local structured editing is trivial because if the braces match, then most of the parser doesn't need to deal with the rest of the file.
- throwaway_pdp09 6y ago
  > of an awful function with multiple screenfuls of code If you've got that, you will have deeper and more abundant problems to solve than will be helped by easily matching up braces.
  - deckard1 6y ago
    heh. That reminds me of JavaScript recently. We have optional chaining now, which quite a few developers absolutely love. Diving deep into data structures is such a code smell, and yet so many people just want to spray the equivalent of Febreze over it. "Maybe with the right syntax for doing this ugly thing people won't notice how ugly this thing I'm doing actually is!"

tomgp 6y ago

If you’re looking for something a little more focused I can heartily recommend Andy Oram and Greg Wilson’s “ Making Software: What Really Works, and Why We Believe It” it’s a great collection of literature reviews around a lots of common software engineering practices.

beprogrammed 6y ago

Wow, what a wonderful pdf.

louiechristie 6y ago

Where’s the ‘conclusions’ section?

RoutinePlayer 6y ago

Evidence-base X is the best X humans can muster.. except for Love

p0nce 6y ago

Honestly sounds very interesting and necessary book.

Geminidog 6y ago

Have you guys ever tried software engineering engineering techniques that utilize No evidence? The current trend of the programming world is to worship data and science but some of the most powerful programming techniques do not involve data or science. A book taking another approach to programming: http://www4.di.uminho.pt/~jno/ps/pdbc.pdf http://www4.di.uminho.pt/~jno/ps/pdbc.pdf Some of you may have trouble distinguishing what is not evidence based and what is evidence based. Hopefully the following dichotomy can clear up any misunderstandings: Science is evidence based, Math is not evidence based. In the real world the ideal approach is to utilize both evidence and non-evidence based techniques, but more often then not the non-evidence based techniques are harder to understand and therefore under utilized.

StonyRhetoric 6y ago

I'm having a hard time deciding if this is the work of a well-meaning but misguided obsessive, or GPT-3 generated pseudo-gibberish. In any case, this book could use a great deal of editing. If there is wisdom in here, I don't want to dig through hundreds of pages to find it.

d23 6y ago
I’m glad I’m not the only one. I started trying to skim it and couldn’t figure out what the hell was going on. Granted, I didn’t give it a strong chance, but the title made it seem like something very different than what it is. I think.

jeff-davis 6y ago

Until we start having a "lines of code" budget, nothing will be fixed. If you want feature X, how many lines of code will it take? 2000? Oh, sorry, we are already spending 6000 lines of code on other features, so find a way to do it in 500 lines or defer the project. Yes, just like anything, this can be gamed. But at least it would get product managers thinking about the complexity they are imposing and the effect that will have on other features.

UK-Al05 6y ago
Sounds like way to form incredibly dense codebases.

arminiusreturns 6y ago

What I appreciate about this is how it addresses the more managerial aspects, things I've been trying hard to wrap my head around so that I can better approach management with things like data based risk analysis based on existing risk analysis algorithms, and things like how to evaluate certain interal functions efficacy. I will very much be reading this thoroughly. Now what I really need is a book on how to break through the ceiling to start communicating with the C's/board better (or at all in some companies where managers and C's are blackholes of information...) This is one of my largest career failures I constantly think about... I see failures that are managerial first but have second tier effects of causing technical issues (technical debt being only one), but everyone in the chain fails to know how to voice this to higher ups, who then tend to assume it's all technical or team based and do things like start replacing managers with PHB's. Sometimes I wish I had gotten an MBA instead just so I could navigate the culuture/speak better. How can an engineer play at the c/board level if he has the desire and aptitude? Any reading recommendations would be appreciated. The older I get the more I regret not "playing politics". Is there a "The Prince - but for SV?"

DoingIsLearning 6y ago
> how to break through the ceiling to start communicating with the C's/board better Anecdata and perhaps unfair but from my own observations, once you get to C-level, who is doing the communicating is much more important than how the message is conveyed. There is definitely a 'club' mentality, usually the way you will get a voice is by building relations with C-levels that will 'adopt' you as their protégé/mentee. Once you have someone in the room who can say "we should listen to this person" then you will have a chance of actually getting info/influence upstream. But to be build those relations there is a fair degree of 'self-promoting' and being seen as the go-getter in the crowd. Silently doing your work and silently going above and beyond is usually just a recipe for disappointment. Another thing I don't often see discussed is that, as with a lot of other things, luck and timing matters a hell of a lot.
- beardedwizard 6y ago
  accurate. People struggle to navigate this realm without being disingenuous though. Stay true, but play the relationship game instead of the write code game.

deleted 6y ago

[deleted]

tdrgabi 6y ago

Hi Derek, thank you for writing the book and releasing it. I haven't seen a similar book before for software engineering, so I'm very intrigued and intend to read it. I feel bad for asking something from a free resource, that took a long time to make and put together, so please ignore it if it's a lot of trouble. Is it possible to publish epub or mobi formats of the book? That would help ereaders format it better, pdf's are inflexible. If it's not easy, please ignore this request. Thank you for writing the book.

osullivj 6y ago

"The craft approach has survived because building software systems has been a sellers market, customers have paid what it takes because the potential benefits have been so much greater than the costs." Gotta disagree here. SW dev has never been a seller's market in the way that eg the property market or job market can be a seller's or buyer's market. There are powerful effects in the market making that a massive simplification. Firstly platform network effects; think of the browser wars for instance. Secondly, open source: a lot can be had for "free". Thirdly, lowest bidder: John Glenn famously commented, when asked what he thought about sat on top of an Apollo rocket, "every component has been built by the lowest bidder". Fourthly, novelty: a significant amount of SW dev is about building things that haven't been built before. Which can be more like a research project than eg putting up a new apartment block, or bridge, or designing and productionising a new car.

woile 6y ago

Is it available in a kindle format? mobi, or epub would be amazing!