6 ms·

I am very skeptical of research about software development. Whether it's about tooling (static typing, etc.) or process (pair programming, etc.). It's really h

I am very skeptical of research about software development. Whether it's about tooling (static typing, etc.) or process (pair programming, etc.). It's really hard (I would go so far as to say impossible) to set up useful metrics and it's really hard to create comparable scenarios. And even if you had those, very few things apply generally to all fields of software, let alone the different types of personalities that developers have, even if the stereotypes have a bit of truth to them.

jcadam 3y ago
Wait, are you suggesting developers are not interchangeable cogs?
- 0xdeadbeefbabe 3y ago
  cogs and sprockets
  - AtlasBarfed 3y ago
    There's pistons and presses in there somewhere as well.
- solardev 3y ago
  Like metric and imperial cogs, software developers also come in different varieties: there are those who prefer tabs, and those who are mistaken.
maerF0x0 3y ago
Wait, so just because there's fuzziness to the data you're going to instead defer to what? What's the proposed alternative/status quo that you're giving a data-less free pass?
- ChadNauseam 3y ago
  You could defer to your personal experience
  - throwaway_08932 3y ago
    Which usually manifests as "your personal comfort."
  - hifromwork 3y ago
    So just data, but with small sample value, biased (since humans are inherently biased) and non filtered by a scientific method. How is that better than proper research? Of course personal experience is very valuable, precisely because it's a (limited form of) research.
    - ChadNauseam 3y ago
      It's not better than proper research, but when proper research doesn't exist it's all we have
      - josephg 3y ago
        Your “gut” instincts can also be a remarkably effective tool in the right situations. You have to feed it enough data and are careful of systematic biases, but your instincts can be brilliant. I had an incredible moment in a job interview: for context, I’ve been programming for about 30 years and 10 of those years have involved a lot of JavaScript. I was given a timed debugging problem to test my skills: “Here is some JS code with failing tests. Go fix the bugs.” Before I even ran the tests I scrolled through the code to get a sense of it. My instincts tweaked on one section. “This code smells. I don’t trust it.” I said. On second glance - “Oh yeah this is totally buggy”. Sure enough, I was right. The interviewer was blown away. There’s no way I could do that consciously. Science is great for a lot of things, but when we don’t have science, we could do a lot worse than trusting our intuition.
    - PH95VuimJjqBqy 3y ago
      some people are better than others, the evidence is there by how some people are more successful than other people. you can attempt to dismiss that as being a small sample value, but that's the reality. The reason people want to be so data driven is because it takes the decision making out of their hands, which takes the responsibility for poor decisions out of their hands. But when it's all said and done, some people are better than others, not all "small sample values" are equal.
- gutnor 3y ago
  Decision driven from bad data is IMO worse than data-less decision. Data grants authority to a decision that gut feel driven one doesn't. It is hard to argue against evidence as it should be, but that assume a certain level of quality in the evidence. Second, if practice doesn't match the expected outcome, the first thing you will look at is what the team is doing wrong, not review the decision as not working. That said, parent is far from unique in his skepticism, so I think the problem is more often reversed in the industry. Having some data, even flawed can help your company decide to try something new.
  - itsoktocry 3y ago
    >Decision driven from bad data is IMO worse than data-less decision. What is "bad data", and how is it worse than no data?
    - mp05 3y ago
      Data gathered from a poorly designed study? I don't think it needs explaining why making decisions on nonsense is just that.
    - tsimionescu 3y ago
      Think of things like "lines of code written" or "bugs closed" as measurements of productivity or quality. These are real things that people have used in real published studies - and any conclusion drawn from them is obviously bogus.
    - itishappy 3y ago
      Measurement noise, other experimental errors, small sample size, cherry picking. It's worse than no data because you can draw incorrect conclusions. Have a really dumb example: All of the developers I work with are short, therefore all developers must be short, and we could save time by filtering job applicants by height. (Note: I work with only one developer. He's great!)
- tikhonj 3y ago
  There is more than some fuzziness to the data! At some point, measuring something sufficiently poorly is worse than not measuring it at all, and the empirical efforts I've seen to evaluate things I care about (programming languages, static types, etc) all fall into that category. The alternative is to rely on rich experience and good taste. If you want to make it a bit more rigorous, you can approach this in terms of qualitative research—which makes sense for academic research, but isn't necessarily the best way to learn for yourself or to design tools. Expert experience is far more effective at capturing complex, multi-dimensional phenomena than anything we can reduce to a small number of easily-gathered quantitative metrics.
  - itsoktocry 3y ago
    >At some point, measuring something sufficiently poorly is worse than not measuring it at all At what point, and how do you determine that? >The alternative is to rely on rich experience and good taste. Everyone thinks they have good taste.
    - worik 3y ago
      > Everyone thinks they have good taste. That is not true In a different domain I play music in bands I have played in a band where the band leader was so so at playing their instrument, but had oodles of taste, so much more than me Not all of us, not many of us, are arrogant pric,s who think we know better or are better than we are
    - tikhonj 3y ago
      > At what point, and how do you determine that? By looking at the research? By understanding what kind of studies and experiments are actually viable? By evaluating what we can measure effectively, what we can't, and how good (or bad!) our proxy metrics can be? As a field, we really need to understand the inherent limits of quantitative methods.
- itsoktocry 3y ago
  >Wait, so just because there's fuzziness to the data you're going to instead defer to what? I'm constantly amazed at how people are willing to throw away data if it's not perfect. All you need is a bit of signal, and it's, literally, better than nothing.
  - riversflow 3y ago
    No, it's not better than nothing. Having played out the exact situation you are describing, If you have some data, but it has no firm causal link between your input and your output, it is simply a distraction and a waste of time and effort to utilize it. You are better off experimenting purely randomly or based on first principles in that case.
- ratherbefuddled 3y ago
  I think fuzziness understates by far the wild variations that uncontrolled (mostly uncontrollable) elements often create. Very few of these kinds of studies follow scientific method well enough to be vaguely useful let alone generally applicable. How many have you ever read of being successfully reproduced?
thaumaturgy 3y ago
Software "engineering" (little to no actual engineering in this area) is still in its infancy. Civil and mechanical engineering are millenia old and we still had the Tacoma Narrows Bridge less than a hundred years ago; electrical engineering is roughly a century old and things still self-ignite on the regular, and there are all kinds of weird and poorly understood quantum thingoes at the really small scales; software engineering is just decades old. Everything that's written on the subject -- every bit of research, fad, and personal anecdote -- is just one more experiment flailing around in the darkness while we try to figure out what it means to "engineer" software. I don't think that makes research about it valueless. Some is good, some is not, some will get revised over time because science and engineering gradually tack towards discovering natural laws. But also, yes, an enormous amount of human effort is wasted on following practices that are often not well-founded.
- egeozcan 3y ago
  > Everything that's written on the subject -- every bit of research, fad, and personal anecdote -- is just one more experiment flailing around in the darkness while we try to figure out what it means to "engineer" software. I think Ian Sommerville does a pretty good job in the book "Software Engineering" (I have the 10th edition), from the section 1.1.1: > Software engineering is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification through to maintaining the system after it has gone into use. In this definition, there are two key phrases: > 1. Engineering discipline: Engineers make things work. They apply theories, methods, and tools where these are appropriate. However, they use them selectively and always try to discover solutions to problems even when there are no applicable theories and methods. Engineers also recognize that they must work within organizational and financial constraints, and they must look for solutions within these constraints. > 2. All aspects of software production: Software engineering is not just concerned with the technical processes of software development. It also includes activities such as software project management and the development of tools, methods, and theories to support software development. These quotes don't do it justice. I'd really recommend this book.
  - thaumaturgy 3y ago
eadler 3y ago
You might enjoy these three things on the topic of software development: 1. Derek Jones. (2021, January 17). Software effort estimation is mostly fake research. The Shape of Code. https://shape-of-code.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://shape-of-code.com/2021/01/17/software-effort-estimat... 2. Andy Oram, Greg Wilson, & Andrew Oram (Eds.). (2011). Making software: what really works, and why we believe it (1st ed). O’Reilly. 3. Bossavit, L. (2015). The leprechauns of software engineering: how folklore turns into fact and what to do about it. Leanpub.

jcadam 3y ago

Wait, are you suggesting developers are not interchangeable cogs?

0xdeadbeefbabe 3y ago
cogs and sprockets
- AtlasBarfed 3y ago
  There's pistons and presses in there somewhere as well.
solardev 3y ago
Like metric and imperial cogs, software developers also come in different varieties: there are those who prefer tabs, and those who are mistaken.

maerF0x0 3y ago

Wait, so just because there's fuzziness to the data you're going to instead defer to what? What's the proposed alternative/status quo that you're giving a data-less free pass?

ChadNauseam 3y ago
You could defer to your personal experience
- throwaway_08932 3y ago
  Which usually manifests as "your personal comfort."
- hifromwork 3y ago
  So just data, but with small sample value, biased (since humans are inherently biased) and non filtered by a scientific method. How is that better than proper research? Of course personal experience is very valuable, precisely because it's a (limited form of) research.
  - ChadNauseam 3y ago
    It's not better than proper research, but when proper research doesn't exist it's all we have
    - josephg 3y ago
      Your “gut” instincts can also be a remarkably effective tool in the right situations. You have to feed it enough data and are careful of systematic biases, but your instincts can be brilliant. I had an incredible moment in a job interview: for context, I’ve been programming for about 30 years and 10 of those years have involved a lot of JavaScript. I was given a timed debugging problem to test my skills: “Here is some JS code with failing tests. Go fix the bugs.” Before I even ran the tests I scrolled through the code to get a sense of it. My instincts tweaked on one section. “This code smells. I don’t trust it.” I said. On second glance - “Oh yeah this is totally buggy”. Sure enough, I was right. The interviewer was blown away. There’s no way I could do that consciously. Science is great for a lot of things, but when we don’t have science, we could do a lot worse than trusting our intuition.
  - PH95VuimJjqBqy 3y ago
    some people are better than others, the evidence is there by how some people are more successful than other people. you can attempt to dismiss that as being a small sample value, but that's the reality. The reason people want to be so data driven is because it takes the decision making out of their hands, which takes the responsibility for poor decisions out of their hands. But when it's all said and done, some people are better than others, not all "small sample values" are equal.
gutnor 3y ago
Decision driven from bad data is IMO worse than data-less decision. Data grants authority to a decision that gut feel driven one doesn't. It is hard to argue against evidence as it should be, but that assume a certain level of quality in the evidence. Second, if practice doesn't match the expected outcome, the first thing you will look at is what the team is doing wrong, not review the decision as not working. That said, parent is far from unique in his skepticism, so I think the problem is more often reversed in the industry. Having some data, even flawed can help your company decide to try something new.
- itsoktocry 3y ago
  >Decision driven from bad data is IMO worse than data-less decision. What is "bad data", and how is it worse than no data?
  - mp05 3y ago
    Data gathered from a poorly designed study? I don't think it needs explaining why making decisions on nonsense is just that.
  - tsimionescu 3y ago
    Think of things like "lines of code written" or "bugs closed" as measurements of productivity or quality. These are real things that people have used in real published studies - and any conclusion drawn from them is obviously bogus.
  - itishappy 3y ago
    Measurement noise, other experimental errors, small sample size, cherry picking. It's worse than no data because you can draw incorrect conclusions. Have a really dumb example: All of the developers I work with are short, therefore all developers must be short, and we could save time by filtering job applicants by height. (Note: I work with only one developer. He's great!)
tikhonj 3y ago
There is more than some fuzziness to the data! At some point, measuring something sufficiently poorly is worse than not measuring it at all, and the empirical efforts I've seen to evaluate things I care about (programming languages, static types, etc) all fall into that category. The alternative is to rely on rich experience and good taste. If you want to make it a bit more rigorous, you can approach this in terms of qualitative research—which makes sense for academic research, but isn't necessarily the best way to learn for yourself or to design tools. Expert experience is far more effective at capturing complex, multi-dimensional phenomena than anything we can reduce to a small number of easily-gathered quantitative metrics.
- itsoktocry 3y ago
  >At some point, measuring something sufficiently poorly is worse than not measuring it at all At what point, and how do you determine that? >The alternative is to rely on rich experience and good taste. Everyone thinks they have good taste.
  - worik 3y ago
    > Everyone thinks they have good taste. That is not true In a different domain I play music in bands I have played in a band where the band leader was so so at playing their instrument, but had oodles of taste, so much more than me Not all of us, not many of us, are arrogant pric,s who think we know better or are better than we are
  - tikhonj 3y ago
    > At what point, and how do you determine that? By looking at the research? By understanding what kind of studies and experiments are actually viable? By evaluating what we can measure effectively, what we can't, and how good (or bad!) our proxy metrics can be? As a field, we really need to understand the inherent limits of quantitative methods.
itsoktocry 3y ago
>Wait, so just because there's fuzziness to the data you're going to instead defer to what? I'm constantly amazed at how people are willing to throw away data if it's not perfect. All you need is a bit of signal, and it's, literally, better than nothing.
- riversflow 3y ago
  No, it's not better than nothing. Having played out the exact situation you are describing, If you have some data, but it has no firm causal link between your input and your output, it is simply a distraction and a waste of time and effort to utilize it. You are better off experimenting purely randomly or based on first principles in that case.
ratherbefuddled 3y ago
I think fuzziness understates by far the wild variations that uncontrolled (mostly uncontrollable) elements often create. Very few of these kinds of studies follow scientific method well enough to be vaguely useful let alone generally applicable. How many have you ever read of being successfully reproduced?

thaumaturgy 3y ago

Software "engineering" (little to no actual engineering in this area) is still in its infancy. Civil and mechanical engineering are millenia old and we still had the Tacoma Narrows Bridge less than a hundred years ago; electrical engineering is roughly a century old and things still self-ignite on the regular, and there are all kinds of weird and poorly understood quantum thingoes at the really small scales; software engineering is just decades old. Everything that's written on the subject -- every bit of research, fad, and personal anecdote -- is just one more experiment flailing around in the darkness while we try to figure out what it means to "engineer" software. I don't think that makes research about it valueless. Some is good, some is not, some will get revised over time because science and engineering gradually tack towards discovering natural laws. But also, yes, an enormous amount of human effort is wasted on following practices that are often not well-founded.

egeozcan 3y ago
> Everything that's written on the subject -- every bit of research, fad, and personal anecdote -- is just one more experiment flailing around in the darkness while we try to figure out what it means to "engineer" software. I think Ian Sommerville does a pretty good job in the book "Software Engineering" (I have the 10th edition), from the section 1.1.1: > Software engineering is an engineering discipline that is concerned with all aspects of software production from the early stages of system specification through to maintaining the system after it has gone into use. In this definition, there are two key phrases: > 1. Engineering discipline: Engineers make things work. They apply theories, methods, and tools where these are appropriate. However, they use them selectively and always try to discover solutions to problems even when there are no applicable theories and methods. Engineers also recognize that they must work within organizational and financial constraints, and they must look for solutions within these constraints. > 2. All aspects of software production: Software engineering is not just concerned with the technical processes of software development. It also includes activities such as software project management and the development of tools, methods, and theories to support software development. These quotes don't do it justice. I'd really recommend this book.
- thaumaturgy 3y ago

eadler 3y ago

You might enjoy these three things on the topic of software development: 1. Derek Jones. (2021, January 17). Software effort estimation is mostly fake research. The Shape of Code. https://shape-of-code.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://shape-of-code.com/2021/01/17/software-effort-estimat... 2. Andy Oram, Greg Wilson, & Andrew Oram (Eds.). (2011). Making software: what really works, and why we believe it (1st ed). O’Reilly. 3. Bossavit, L. (2015). The leprechauns of software engineering: how folklore turns into fact and what to do about it. Leanpub.