8 ms·

Program comprehension is a fascinating (and IMHO very enlightening) field. For anyone curious, I wrote up some of what I learned while exploring the research a

Program comprehension is a fascinating (and IMHO very enlightening) field. For anyone curious, I wrote up some of what I learned while exploring the research a few years ago: http://www.clarityincode.com/readability/ http://www.clarityincode.com/readability/ (I apologise for the less than stellar formatting; I haven’t updated that page for some time. I also apologise to the actual researchers I cited, if I’ve dumbed down their work too much in aiming for a non-expert audience.) For the eye tracking reported here, I wonder whether the early emphasis on the top part was a combination of trying to figure out the data flow in the between() function and then its significance to the wider program. I think it would be interesting to compare the results with a similar eye track of a program written with more emphasis on data flow rather than control flow, e.g., def between(numbers, low, high): return [n for n in numbers if low < n < high] def common(list1, list2): return [i for i in list1 if i in list2] x = [2, 8, 7, 9, -5, 0, 2] x_btwn = between(x, 2, 10) print x_btwn y = [1, -3, 10, 0, 8, 9, 1] y_btwn = between(y, -2, 9) print y_btwn xy_common = common(x, y) print xy_common It might also be interesting to compare the results with a functional programming language that expresses those ideas more concisely and/or with tools like between() and common() as part of the standard library that programmers would probably be familiar with. Final thought: How much does the absence of a clearly marked starting point (like a main() function in C) affect how a reader approaches unknown code in Python? If this had been a C program, would the reader have aimed straight for main() and then worked down from there to functions like between() and common()?

dchichkov 14y ago
I think it is very important to use code from real-life applications, rather then synthetic examples like that, or like ones presented in the original article. Otherwise the results of that research would be biased towards comprehension of meaningless chunks of code, rather that of a real thing.
- Chris_Newton 14y ago
  I agree, but isn’t this the curse of most experimental research in programming techniques? Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach. What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?
  - saraid216 14y ago
    This actually smells like the same kind of problem you'd have in studying classrooms...
    - Chris_Newton 14y ago
      I’m not sure if this is what you were alluding to there, but I think there is a real problem with experimental research in this kind of field that often the test subjects are students, who tend to have much less skill and experience than working programmers with a few years of practice behind them. It’s a specific instance of a more general problem: it seems clear that programmers often work differently depending on their familiarity both with programming generally and with the specific domain they’re working in, but we’ve only scratched the surface in identifying exactly how they work differently, and therefore what practical steps we might take to make things better for programmers in different situations.
      - infinite8s 14y ago
        This is often the problem with a lot of psychology research, where the only easily available subjects are university students who receive credit for participating in the research (or a nominal payment).
        synesthesiam 14y ago
        For some kinds of experiments, Mechanical Turk is helping to alleviate the problem: http://pps.sagepub.com/content/6/1/3.short http://pps.sagepub.com/content/6/1/3.short
  - dchichkov 14y ago
    Yes, but why can't some regular code (from an open sourced project) be used in the experiments? I'm sure there should be a way to do that. It is just, that this synthetic code is broken. It doesn't read. There is no flow in it. It just looks from the first glance, like a non-interesting, unimportant piece, that doesn't do anything, so there is no point in reading it. Does that sound convincing enough, that there is a big difference between reading synthetic examples and actual code?
- eholk 14y ago
  For an experiment like this it's important to have small, self contained code fragments. It wouldn't be fair to expect someone to decipher the whole Linux kernel source in a short amount of time, for example. Given a sufficiently small chunk of code, almost everything looks synthetic. I imagine many real life application have sections that take two arrays, find the numbers within a certain range and then take the intersection like this program does.
  - dchichkov 14y ago
    Actually linux kernel is very readable. And there is nothing synthetic about it. Just try opening some random page of linux kernel source code, and you'll see. Say: http://www.linux.fm/source/crypto/xor.c/ http://www.linux.fm/source/crypto/xor.c/ Hey, you can even listen to it ;)
- scott_s 14y ago
  That's step 2. Step 1 is using contrived examples with clear differences in structure. You need to start small like this because in order to answer the basic questions, you need to eliminate experimental variables. That way, you can establish basic principles for how people tend to build up an understanding of code. Then you can move on to what we're really interested in: how do people build an understanding of complex applications with many interacting parts? But you can't go there first; if you do, you won't be able to interpret your results because there will be just as many fundamental questions as experimental variables.
Kutta 14y ago
This seems to be massively more readable to me. Looking at the video version of the code I spent quite a few seconds looking at the non-chained comparison ("everyone chains comparison when checking bounds. Why isn't it that way here?"), and the "winner" containers (is there something winning something?).
- Chris_Newton 14y ago
  FWIW, the most important conclusions from exploring the research that I’ve drawn personally are: 1. Establishing familiar patterns is good, and being able to identify deviations from patterns (intentional or otherwise) is probably far more important than we usually assume. 2. You’re going to need to figure out the data flow for some code you’re reading anyway, and a lot of the time the control flow is just noise/accidental complexity, so why not put the data flow front and centre if possible? Applying these to the example code we’ve been discussing, I might prefer to write it something like this: x = [2, 8, 7, 9, -5, 0, 2] y = [1, -3, 10, 0, 8, 9, 1] print filter ( 2 < _ < 10) x print filter (-2 < _ < 9) y print intersection x y with the proviso that the previous common() function doesn't quite do what an intersection() usually would, which in itself might prompt us to think about whether the previous common() function that is asymmetric in its parameters is actually what we wanted. (Maybe it was, or maybe it was an unintentional error but no-one noticed because the results are the same with this set of sample data.) I have a theory that part of the attraction/intrigue of functional programming for a lot of working programmers comes from the way it inherently emphasizes these ideas, particularly making data flow almost the default form of presentation. I also have a theory that maybe there's another mental model to be found/learned/created, somewhere above control flow and data flow but below the general domain model, capturing the flow of observable interactions with the outside world: side effects, interactions/dependencies between concurrent operations, and so on. I suspect that part of the reason we find functional programming more difficult for some things (see the “awkward squad” paper) is that when you remove the implicit control flow, you also remove the implicit ordering of events/effects that you get with imperative programming, and there is a cost to that if you don’t replace it (or at least the parts of it that matter for that higher-level view of what your code is doing) with something else.
drothlis 14y ago
> I think it would be interesting to compare the results with a similar eye track of a program written with more emphasis on data flow It would also be interesting to compare with code with decent syntax highlighting. For some reason I don't quite understand, most syntax highlighting out there highlights the wrong thing: keywords. It needs to highlight the site-of-definition of functions, variables, etc. For example: http://david.rothlis.net/code_presentation/distracting_syntax_highlighting/ http://david.rothlis.net/code_presentation/distracting_synta...
- kibwen 14y ago
  It's pretty easy to understand why keywords are highlighted, even if it isn't justifiable. 1) Programmers are lazy, 2) regexes are the quickest way to implement basic syntax highlighting, and 3) if your keywords are reserved words (as most are), then you can get infallible highlighting with the expression /(^|\s):listofreservedwords:(/s|$)/.
  - drothlis 14y ago
    Emacs, at least, already provides primitive parsing / heuristics that work reasonably well for most languages. See the example in the link I posted above: All I had to do was customise the colour and font-weight of various things already recognised by Emacs: function-name, variable-name, type, etc. Maybe what I don't understand is all the effort people put into making pretty themes for their editors --there are millions of these themes out there-- and almost all of them insist on highlighting (drawing attention to) the language's keywords of all things. The least important part of any program.
- eholk 14y ago
  Some of the initial analysis Mike has done shows that programmers spend the majority of their time looking at identifiers. I think this supports your point.
- synesthesiam 14y ago
  This is something I'm definitely interested in. I don't have any syntax highlighting in the current experiment to avoid controversy over the "right" way to highlight, and to put the focus on content of the code. For the future, though, this would be an excellent experimental manipulation to try. Thanks!

dchichkov 14y ago

I think it is very important to use code from real-life applications, rather then synthetic examples like that, or like ones presented in the original article. Otherwise the results of that research would be biased towards comprehension of meaningless chunks of code, rather that of a real thing.

Chris_Newton 14y ago
I agree, but isn’t this the curse of most experimental research in programming techniques? Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach. What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?
- saraid216 14y ago
  This actually smells like the same kind of problem you'd have in studying classrooms...
  - Chris_Newton 14y ago
    I’m not sure if this is what you were alluding to there, but I think there is a real problem with experimental research in this kind of field that often the test subjects are students, who tend to have much less skill and experience than working programmers with a few years of practice behind them. It’s a specific instance of a more general problem: it seems clear that programmers often work differently depending on their familiarity both with programming generally and with the specific domain they’re working in, but we’ve only scratched the surface in identifying exactly how they work differently, and therefore what practical steps we might take to make things better for programmers in different situations.
    - infinite8s 14y ago
      This is often the problem with a lot of psychology research, where the only easily available subjects are university students who receive credit for participating in the research (or a nominal payment).
      - synesthesiam 14y ago
        For some kinds of experiments, Mechanical Turk is helping to alleviate the problem: http://pps.sagepub.com/content/6/1/3.short http://pps.sagepub.com/content/6/1/3.short
- dchichkov 14y ago
  Yes, but why can't some regular code (from an open sourced project) be used in the experiments? I'm sure there should be a way to do that. It is just, that this synthetic code is broken. It doesn't read. There is no flow in it. It just looks from the first glance, like a non-interesting, unimportant piece, that doesn't do anything, so there is no point in reading it. Does that sound convincing enough, that there is a big difference between reading synthetic examples and actual code?
eholk 14y ago
For an experiment like this it's important to have small, self contained code fragments. It wouldn't be fair to expect someone to decipher the whole Linux kernel source in a short amount of time, for example. Given a sufficiently small chunk of code, almost everything looks synthetic. I imagine many real life application have sections that take two arrays, find the numbers within a certain range and then take the intersection like this program does.
- dchichkov 14y ago
  Actually linux kernel is very readable. And there is nothing synthetic about it. Just try opening some random page of linux kernel source code, and you'll see. Say: http://www.linux.fm/source/crypto/xor.c/ http://www.linux.fm/source/crypto/xor.c/ Hey, you can even listen to it ;)
scott_s 14y ago
That's step 2. Step 1 is using contrived examples with clear differences in structure. You need to start small like this because in order to answer the basic questions, you need to eliminate experimental variables. That way, you can establish basic principles for how people tend to build up an understanding of code. Then you can move on to what we're really interested in: how do people build an understanding of complex applications with many interacting parts? But you can't go there first; if you do, you won't be able to interpret your results because there will be just as many fundamental questions as experimental variables.

Kutta 14y ago

This seems to be massively more readable to me. Looking at the video version of the code I spent quite a few seconds looking at the non-chained comparison ("everyone chains comparison when checking bounds. Why isn't it that way here?"), and the "winner" containers (is there something winning something?).

Chris_Newton 14y ago
FWIW, the most important conclusions from exploring the research that I’ve drawn personally are: 1. Establishing familiar patterns is good, and being able to identify deviations from patterns (intentional or otherwise) is probably far more important than we usually assume. 2. You’re going to need to figure out the data flow for some code you’re reading anyway, and a lot of the time the control flow is just noise/accidental complexity, so why not put the data flow front and centre if possible? Applying these to the example code we’ve been discussing, I might prefer to write it something like this: x = [2, 8, 7, 9, -5, 0, 2] y = [1, -3, 10, 0, 8, 9, 1] print filter ( 2 < _ < 10) x print filter (-2 < _ < 9) y print intersection x y with the proviso that the previous common() function doesn't quite do what an intersection() usually would, which in itself might prompt us to think about whether the previous common() function that is asymmetric in its parameters is actually what we wanted. (Maybe it was, or maybe it was an unintentional error but no-one noticed because the results are the same with this set of sample data.) I have a theory that part of the attraction/intrigue of functional programming for a lot of working programmers comes from the way it inherently emphasizes these ideas, particularly making data flow almost the default form of presentation. I also have a theory that maybe there's another mental model to be found/learned/created, somewhere above control flow and data flow but below the general domain model, capturing the flow of observable interactions with the outside world: side effects, interactions/dependencies between concurrent operations, and so on. I suspect that part of the reason we find functional programming more difficult for some things (see the “awkward squad” paper) is that when you remove the implicit control flow, you also remove the implicit ordering of events/effects that you get with imperative programming, and there is a cost to that if you don’t replace it (or at least the parts of it that matter for that higher-level view of what your code is doing) with something else.

drothlis 14y ago

> I think it would be interesting to compare the results with a similar eye track of a program written with more emphasis on data flow It would also be interesting to compare with code with decent syntax highlighting. For some reason I don't quite understand, most syntax highlighting out there highlights the wrong thing: keywords. It needs to highlight the site-of-definition of functions, variables, etc. For example: http://david.rothlis.net/code_presentation/distracting_syntax_highlighting/ http://david.rothlis.net/code_presentation/distracting_synta...

kibwen 14y ago
It's pretty easy to understand why keywords are highlighted, even if it isn't justifiable. 1) Programmers are lazy, 2) regexes are the quickest way to implement basic syntax highlighting, and 3) if your keywords are reserved words (as most are), then you can get infallible highlighting with the expression /(^|\s):listofreservedwords:(/s|$)/.
- drothlis 14y ago
  Emacs, at least, already provides primitive parsing / heuristics that work reasonably well for most languages. See the example in the link I posted above: All I had to do was customise the colour and font-weight of various things already recognised by Emacs: function-name, variable-name, type, etc. Maybe what I don't understand is all the effort people put into making pretty themes for their editors --there are millions of these themes out there-- and almost all of them insist on highlighting (drawing attention to) the language's keywords of all things. The least important part of any program.
eholk 14y ago
Some of the initial analysis Mike has done shows that programmers spend the majority of their time looking at identifiers. I think this supports your point.
synesthesiam 14y ago
This is something I'm definitely interested in. I don't have any syntax highlighting in the current experiment to avoid controversy over the "right" way to highlight, and to put the focus on content of the code. For the future, though, this would be an excellent experimental manipulation to try. Thanks!