4 ms·

Bleh. I think the software development industry should stop doing such a disservice to itself, and start funding more scientific studies in software developmen

Bleh. I think the software development industry should stop doing such a disservice to itself, and start funding more scientific studies in software development. Random, almost shower-thought hypotheses like 'naming things is hard - we should stop doing it' are given way too much credibility with nothing to back them up.

braythwayt 11y ago
> the software development industry should stop doing such a disservice to > itself, and start funding more scientific studies in software > development. Yes. > hypotheses like 'naming things is hard - we should stop doing it' are given > way too much credibility with nothing to back them up. Also yes, but let's separate generating hypotheses and discussing them from giving them credibility. It's a net win for people to share their shower-time hypotheses with each other. It's a net win for sites like Hacker News to feature them so that the "long tail of thought" can stretch ever longer. But as you say, we need to keep some perspective and grant that these are hypotheses. For example, a lot of times, people try turning a dial to eleven: Long iterations are hard, what's the shortest iteration cycle? What if we try releasing every two weeks? Code review is valuable, what if we review 24/7/365 by pairing or even mob programming? Long methods are hard to understand, what if we make a rule that they can be no longer than five lines? And so forth. And they get good n=1 anecdotal results. My hypothesis is that there's a Hawthorne Effect going on: When you turn a dial to eleven, you find yourself forced to observe and think about what's going on very thoroughly, and in the process of doing so, you get better results. Then you keep doing that, and eventually the new way of doing things is internalized, and you stop being forced to think very hard about things, and you regress to your old level of hand-wavey productivity/quality/whatever. This little pet model of mine explains why things like this keep popping up, why they are often valuable exercises, but also why, as you say, we should do some real measuring before pronouncing that we're DoingItWrong™.
seiji 11y ago
more scientific studies in software development. There's an entire field called Software Engineering. But, it's more fun to just do what feels quick and satisfying instead of binding yourself to processes other people dictated.
- thearn4 11y ago
  > Software Engineering Unfortunately, I feel like that term has been very diluted in the last few years, to the point where I'm not sure it has much difference (at least colloquially) from "programming".
  - seiji 11y ago
    Well, it's typically a multi-year degree program and not a 300 word blog post. https://en.wikipedia.org/wiki/Software_engineering https://en.wikipedia.org/wiki/Software_engineering Then you have Agile and XP and Scrum as either fast-flux Software Engineering (without all the "formal process/documentation" and instead use informal, but blessed, processes and immediate, undocumented, decision making) or anti-software-engineering depending on how you look at it. Think: space shuttle or airliners, not social media. Sure, they still aren't perfect, but if you told me my airplane ran on Ruby I'd rather walk to my destination.
    - innguest 11y ago
      It's also typically not an Engineering degree so maybe we should stop calling it that.
      - seiji 11y ago
        You are correct, but it's still the name of an entire field of study. When things have names, we have to use the names the things have when talking about them. The initial page for it on wikiepdia is interesting too (November 14, 2001): https://en.wikipedia.org/w/index.php?title=Software_engineering&oldid=334847426 https://en.wikipedia.org/w/index.php?title=Software_engineer... Especially: Methods have improved through the 1980s and 1990s, but the complexity in requirements have increased fast enough to consume any improvement. There is also a constant eagerness to develop new tools and try the newest tools in each new project. If the programmers are good, the system will work, and the customer will not fully appreciate how complex the task was.
    - ebiester 11y ago
lsc 11y ago
>I think the software development industry should stop doing such a disservice to itself, and start funding more scientific studies in software development. Hm. I wonder if some of the larger companies are doing that sort of thing and keeping it internal-only or using it for internal coding standards without publishing it in an attempt to gain competitive advantage?
- bunderbunder 11y ago
  I doubt it. And even if they did, I don't think you should expect their results to generalize. There's some evidence to suggest that the reason why studies like this so often produce seemingly conflicting results is that there aren't very many universal rules. The same practice can be helpful for one team and harmful for another based on any number of circumstances.
  - tetha 11y ago
    It's even in the same company. My team has very few common problems with our product teams. In fact, some of our experiences are just contrary to the product teams experiences. Overall, software development is looking more and more like a journey to me. And a journey in the polar sea is different from the journey in the sahara. Just don't leave jenkins at home in either case.
    - lsc 11y ago
      eh, I think that there is a lot of value to standardization. Your average programmer is gonna spend more time reading code than writing it; there's something to be said for getting used to one standard style.
eccles 11y ago
Hear hear. Computer Science mixed with human factors looks similar to psychology in the early 20th century (we clearly have mommy problems). The statement "there are only 2 hard problems" is cute and pithy but doesn't really match my experience over the last 15 years. Most of my painful memories relate to: multi-threading, timezones (bloody timezones), migration a system from A to B tech (i.e. SQL to NoSQL), ensuring a distributed system does something once--and only once, refactoring a god object, delegating a problem to an uneven strength team, building a system where the team organization doesn't match the desired architecture, getting the right requirements, dependency management, and deployment. There are probably more but that's beside the point. There is research to support some of my observations but it's hidden behind paywalls or has thin supporting evidence. Comprehensive studies are hard and a lot of the research uses only students as convenience subjects. There really isn't a strong connection between the research community and practice in order to provide the right measurements. What I would like to see is adding support to GitHub for community research studies. There is wealth of pure code observations. It would be interesting to allow integration of survey tools and marry that to the code observations. It'll answer some of the problems but not others (i.e. the pain of maintaining a live service). And finally as part of the GitHub EULA this research must be publicly available.
ebiester 11y ago
Check out the work around http://quorumlanguage.com/ http://quorumlanguage.com/ -- it turns out that these are harder to quantify than we would expect. For example, functional programming is terrible for beginners and has a high learning curve. People who use it in production swear by it. How do we test this? Can we trust that what is true for a beginner is true for an expert? How do you afford 10 mid-level programmers for 6 months? Is that a large enough sample size? The fundamental problem with studies, as I see it, is that we haven't figured out how to constrain the variability yet. We have a lot of work to get out of the hypothesis stage still.
fiatjaf 11y ago
Scientific? Bleh.

braythwayt 11y ago

> the software development industry should stop doing such a disservice to > itself, and start funding more scientific studies in software > development. Yes. > hypotheses like 'naming things is hard - we should stop doing it' are given > way too much credibility with nothing to back them up. Also yes, but let's separate generating hypotheses and discussing them from giving them credibility. It's a net win for people to share their shower-time hypotheses with each other. It's a net win for sites like Hacker News to feature them so that the "long tail of thought" can stretch ever longer. But as you say, we need to keep some perspective and grant that these are hypotheses. For example, a lot of times, people try turning a dial to eleven: Long iterations are hard, what's the shortest iteration cycle? What if we try releasing every two weeks? Code review is valuable, what if we review 24/7/365 by pairing or even mob programming? Long methods are hard to understand, what if we make a rule that they can be no longer than five lines? And so forth. And they get good n=1 anecdotal results. My hypothesis is that there's a Hawthorne Effect going on: When you turn a dial to eleven, you find yourself forced to observe and think about what's going on very thoroughly, and in the process of doing so, you get better results. Then you keep doing that, and eventually the new way of doing things is internalized, and you stop being forced to think very hard about things, and you regress to your old level of hand-wavey productivity/quality/whatever. This little pet model of mine explains why things like this keep popping up, why they are often valuable exercises, but also why, as you say, we should do some real measuring before pronouncing that we're DoingItWrong™.

seiji 11y ago

more scientific studies in software development. There's an entire field called Software Engineering. But, it's more fun to just do what feels quick and satisfying instead of binding yourself to processes other people dictated.

thearn4 11y ago
> Software Engineering Unfortunately, I feel like that term has been very diluted in the last few years, to the point where I'm not sure it has much difference (at least colloquially) from "programming".
- seiji 11y ago
  Well, it's typically a multi-year degree program and not a 300 word blog post. https://en.wikipedia.org/wiki/Software_engineering https://en.wikipedia.org/wiki/Software_engineering Then you have Agile and XP and Scrum as either fast-flux Software Engineering (without all the "formal process/documentation" and instead use informal, but blessed, processes and immediate, undocumented, decision making) or anti-software-engineering depending on how you look at it. Think: space shuttle or airliners, not social media. Sure, they still aren't perfect, but if you told me my airplane ran on Ruby I'd rather walk to my destination.
  - innguest 11y ago
    It's also typically not an Engineering degree so maybe we should stop calling it that.
    - seiji 11y ago
      You are correct, but it's still the name of an entire field of study. When things have names, we have to use the names the things have when talking about them. The initial page for it on wikiepdia is interesting too (November 14, 2001): https://en.wikipedia.org/w/index.php?title=Software_engineering&oldid=334847426 https://en.wikipedia.org/w/index.php?title=Software_engineer... Especially: Methods have improved through the 1980s and 1990s, but the complexity in requirements have increased fast enough to consume any improvement. There is also a constant eagerness to develop new tools and try the newest tools in each new project. If the programmers are good, the system will work, and the customer will not fully appreciate how complex the task was.
  - ebiester 11y ago

lsc 11y ago

>I think the software development industry should stop doing such a disservice to itself, and start funding more scientific studies in software development. Hm. I wonder if some of the larger companies are doing that sort of thing and keeping it internal-only or using it for internal coding standards without publishing it in an attempt to gain competitive advantage?

bunderbunder 11y ago
I doubt it. And even if they did, I don't think you should expect their results to generalize. There's some evidence to suggest that the reason why studies like this so often produce seemingly conflicting results is that there aren't very many universal rules. The same practice can be helpful for one team and harmful for another based on any number of circumstances.
- tetha 11y ago
  It's even in the same company. My team has very few common problems with our product teams. In fact, some of our experiences are just contrary to the product teams experiences. Overall, software development is looking more and more like a journey to me. And a journey in the polar sea is different from the journey in the sahara. Just don't leave jenkins at home in either case.
  - lsc 11y ago
    eh, I think that there is a lot of value to standardization. Your average programmer is gonna spend more time reading code than writing it; there's something to be said for getting used to one standard style.

eccles 11y ago

Hear hear. Computer Science mixed with human factors looks similar to psychology in the early 20th century (we clearly have mommy problems). The statement "there are only 2 hard problems" is cute and pithy but doesn't really match my experience over the last 15 years. Most of my painful memories relate to: multi-threading, timezones (bloody timezones), migration a system from A to B tech (i.e. SQL to NoSQL), ensuring a distributed system does something once--and only once, refactoring a god object, delegating a problem to an uneven strength team, building a system where the team organization doesn't match the desired architecture, getting the right requirements, dependency management, and deployment. There are probably more but that's beside the point. There is research to support some of my observations but it's hidden behind paywalls or has thin supporting evidence. Comprehensive studies are hard and a lot of the research uses only students as convenience subjects. There really isn't a strong connection between the research community and practice in order to provide the right measurements. What I would like to see is adding support to GitHub for community research studies. There is wealth of pure code observations. It would be interesting to allow integration of survey tools and marry that to the code observations. It'll answer some of the problems but not others (i.e. the pain of maintaining a live service). And finally as part of the GitHub EULA this research must be publicly available.

ebiester 11y ago

Check out the work around http://quorumlanguage.com/ http://quorumlanguage.com/ -- it turns out that these are harder to quantify than we would expect. For example, functional programming is terrible for beginners and has a high learning curve. People who use it in production swear by it. How do we test this? Can we trust that what is true for a beginner is true for an expert? How do you afford 10 mid-level programmers for 6 months? Is that a large enough sample size? The fundamental problem with studies, as I see it, is that we haven't figured out how to constrain the variability yet. We have a lot of work to get out of the hypothesis stage still.

fiatjaf 11y ago

Scientific? Bleh.