3 ms·

Isn't this a fundamental difficulty with any flat text-file representation for code, though? Many programming languages don't lend themselves to easy semantic a

Isn't this a fundamental difficulty with any flat text-file representation for code, though? Many programming languages don't lend themselves to easy semantic analysis based on even a single file, never mind a combination of files. We usually don't work with canonical textual representations of even quite simple programming ideas such as the example in the blog post of replacing a nested condition with a guard clause. At our current level of expressiveness, the refactoring tools that make these semantic changes mostly aren't a huge step up from brute force text editing, and the mechanical changes we can automate are usually trivial compared to the way a developer might conceive a required change in the behaviour of his code. Trying to reverse the process to identify the semantic significance of changes after the fact is far beyond us today. In fact, it's hard to see how we could ever move beyond that level without developing some new and much more semantically rich way to represent our programs. At that point, the idea of diffing raw text code side-by-side might seem like it comes from the dark ages anyway...

marshray 15y ago
There have been a thousand graduate school and commercial project to try to develop program representations and languages that were superior to the flat text file. Obviously, none of them have received wide usage. What has succeeded for many is scriptable text editors, cscope-like tools, autocomplete-like features, and refactoring support in editors. But as you point out, these are better tools to work with program text. I think text is safe for the forseeable future. Partly because we have thousands of years experience with it and partly because it's a winning combination of a extremely powerful representation format and a KISS solution.
- Chris_Newton 15y ago
  I agree that, for better or worse, we'll be using text-based formats for a while. I don't think the hypothetical alternative representation I mentioned is going to happen off the back of a single grad school research project, or anything even close to that scale. It's more like something that's going to take the R&D lab of an industry heavweight a decade to develop and refine and then launch into widespread industrial usage with the backing of at least one major platform developer when it's ready for prime time. Sadly, as long as text-based (or, perhaps more accurately, line-based) programming languages are good enough to produce acceptable software, there isn't a truly compelling motivation to develop a completely new model. I wonder whether increasing pressure on the software industry to provide quality and security as a backlash against the current cheap-and-nasty trend will drive us toward more radical programming models first, and perhaps get us close enough to make the quantum leap to an entirely new kind of representation from there.
- stcredzero 15y ago
  There have been a thousand graduate school and commercial project to try to develop program representations and languages that were superior to the flat text file. Obviously, none of them have received wide usage. Chunk format Smalltalk change logs are a text file, but they're combined with a mechanism for treating code changes like db transaction logs, in a language with very little syntax, with very fine grained codebase change (method level, with generally small methods) built into the language and environment. It's also as rock solid as traditional flat text files, and even more robust in some regards. Code syntax which requires all information relevant in a class to be defined together interferes with a change log scheme. But such syntactic structures are helpful if dealing with code in traditional flat files. In a way, it's like an Evolutionary Stable Strategy. It's mostly a flat file world, so the languages are mostly implemented with that in mind. This general situation makes it almost impossible for anything else to develop its won ecosystem. I think forward progress will be made by leveraging "code folding" schemes. Eventually, this will amount to the same functionality people are aiming for then they discuss non-text representations.
  - stcredzero 15y ago
    Arrgh: when, not then. Thanks again iPad!
sliverstorm 15y ago
You'd probably be better off working on the revision control side of things. As a simple example, develop a more semantically rich way to represent the diff for human review. Perhaps a tree graph in both panes with highlighting based on what has changed, as well as nodes that will be affected by the change (e.g. functions that call the changed function) I say this because such a tool may be easily adopted into existing workflows. This is key.
tsewlliw 15y ago
IMHO the limiting factor can be described as "semantic density". If you look at non-LCS diff algorithms like patience diff [1] and think about where it falls down, its on cases where its very arguable that the code is poor - too much duplication therefore low semantic density. I think (and am attempting to further this at work for non-code) that the effective key is to do something like patience diff on characters but looking at the context of the individual characters when determining whether they are unique. The benefit here is that you don't need to know about the parse tree, don't depend on special characters to delimit elements (newlines), and don't get LCS type diffs that aren't based on any sort of semantic intuition beyond that the minimal change is probably what was meant. [1] http://bramcohen.livejournal.com/73318.html http://bramcohen.livejournal.com/73318.html
- cpeterso 15y ago
  I've read about Patience diff, but could not find good, real-world examples where LCS diff choked but Patience succeeded. Just last week, however, I had a small code change that produced a crazy diff that would give my code reviewer an unnecessary headache. I tried Patience diff on a whim and it produced a clean diff. Admittedly, it just entirely -removed the old code block and +added the new one, but at least it was readable.
  - tsewlliw 15y ago
    I work on a code review tool, and we've been shipping patience diff for a few years now, and it has really cut down on complaints about the resulting diff being incomprehensible, and on the one or two occasions people have complained its been really easy to point at the data explain why - and that has always been because there was a "missed opportunity to write better code" or a big pile of copy-pasta. I know its sort of a silly argument to make, but LCS's algorithmic intiution leaves me cold.
    - cpeterso 15y ago
      Very interesting. Is patience diff your tool's default algorithm? Do you support diff algorithms (perhaps for different file types) other than patience or LCS?

marshray 15y ago

There have been a thousand graduate school and commercial project to try to develop program representations and languages that were superior to the flat text file. Obviously, none of them have received wide usage. What has succeeded for many is scriptable text editors, cscope-like tools, autocomplete-like features, and refactoring support in editors. But as you point out, these are better tools to work with program text. I think text is safe for the forseeable future. Partly because we have thousands of years experience with it and partly because it's a winning combination of a extremely powerful representation format and a KISS solution.

Chris_Newton 15y ago
I agree that, for better or worse, we'll be using text-based formats for a while. I don't think the hypothetical alternative representation I mentioned is going to happen off the back of a single grad school research project, or anything even close to that scale. It's more like something that's going to take the R&D lab of an industry heavweight a decade to develop and refine and then launch into widespread industrial usage with the backing of at least one major platform developer when it's ready for prime time. Sadly, as long as text-based (or, perhaps more accurately, line-based) programming languages are good enough to produce acceptable software, there isn't a truly compelling motivation to develop a completely new model. I wonder whether increasing pressure on the software industry to provide quality and security as a backlash against the current cheap-and-nasty trend will drive us toward more radical programming models first, and perhaps get us close enough to make the quantum leap to an entirely new kind of representation from there.
stcredzero 15y ago
There have been a thousand graduate school and commercial project to try to develop program representations and languages that were superior to the flat text file. Obviously, none of them have received wide usage. Chunk format Smalltalk change logs are a text file, but they're combined with a mechanism for treating code changes like db transaction logs, in a language with very little syntax, with very fine grained codebase change (method level, with generally small methods) built into the language and environment. It's also as rock solid as traditional flat text files, and even more robust in some regards. Code syntax which requires all information relevant in a class to be defined together interferes with a change log scheme. But such syntactic structures are helpful if dealing with code in traditional flat files. In a way, it's like an Evolutionary Stable Strategy. It's mostly a flat file world, so the languages are mostly implemented with that in mind. This general situation makes it almost impossible for anything else to develop its won ecosystem. I think forward progress will be made by leveraging "code folding" schemes. Eventually, this will amount to the same functionality people are aiming for then they discuss non-text representations.
- stcredzero 15y ago
  Arrgh: when, not then. Thanks again iPad!

sliverstorm 15y ago

You'd probably be better off working on the revision control side of things. As a simple example, develop a more semantically rich way to represent the diff for human review. Perhaps a tree graph in both panes with highlighting based on what has changed, as well as nodes that will be affected by the change (e.g. functions that call the changed function) I say this because such a tool may be easily adopted into existing workflows. This is key.

tsewlliw 15y ago

IMHO the limiting factor can be described as "semantic density". If you look at non-LCS diff algorithms like patience diff [1] and think about where it falls down, its on cases where its very arguable that the code is poor - too much duplication therefore low semantic density. I think (and am attempting to further this at work for non-code) that the effective key is to do something like patience diff on characters but looking at the context of the individual characters when determining whether they are unique. The benefit here is that you don't need to know about the parse tree, don't depend on special characters to delimit elements (newlines), and don't get LCS type diffs that aren't based on any sort of semantic intuition beyond that the minimal change is probably what was meant. [1] http://bramcohen.livejournal.com/73318.html http://bramcohen.livejournal.com/73318.html

cpeterso 15y ago
I've read about Patience diff, but could not find good, real-world examples where LCS diff choked but Patience succeeded. Just last week, however, I had a small code change that produced a crazy diff that would give my code reviewer an unnecessary headache. I tried Patience diff on a whim and it produced a clean diff. Admittedly, it just entirely -removed the old code block and +added the new one, but at least it was readable.
- tsewlliw 15y ago
  I work on a code review tool, and we've been shipping patience diff for a few years now, and it has really cut down on complaints about the resulting diff being incomprehensible, and on the one or two occasions people have complained its been really easy to point at the data explain why - and that has always been because there was a "missed opportunity to write better code" or a big pile of copy-pasta. I know its sort of a silly argument to make, but LCS's algorithmic intiution leaves me cold.
  - cpeterso 15y ago
    Very interesting. Is patience diff your tool's default algorithm? Do you support diff algorithms (perhaps for different file types) other than patience or LCS?