5 ms·

I’m almost not sure why tools like git don’t ship with this as default. Been using difft for about a year now, and my main complaint is that it makes it hard to

I’m almost not sure why tools like git don’t ship with this as default. Been using difft for about a year now, and my main complaint is that it makes it hard to go back and use other diff tools when I don’t have difft available :). I am curious if there’s been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same). It seems like an intractable problem in the general but maybe it’s doable and/or useful for smaller DSLs or subsets of some languages?

ruined 3y ago
>I am curious if there’s been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same). if you do this your difftool becomes a compiler
- hobs 3y ago
  That's exactly what I have done with diffing SQL in lazy mode - just use a server and diff the AST/plan.
  - slotrans 3y ago
    Two semantically equivalent SQL statements can plan differently...
    - rrrrrrrrrrrryan 3y ago
      The exact same SQL statement can plan differently if table statistics change.
    - hobs 3y ago
      Absolutely and for this case a different plan mattered.
- mlavrent 3y ago
  Sorry, I should've been clearer. I'm interested if there's any tool that does this kind of thing statically, without running the code. I guess a simple approach is to compile both programs and see if the generated code is the same, but I'd guess reasoning at the generated-code level will probably produce a lot more false positives (i.e. tool will report a change when there isn't one) than if you reason about the original program.
  - jerf 3y ago
    This gets really hard, really fast. That is, yes, reasonably obviously doing this completely 100% accurately requires a solution to the halting problem, but even getting to "useful" is really really hard. Even the Haskell world doesn't try to solve the "equivalence of functions" problem, and it's even more complicated in imperative languages. You probably have a mental image of catching something really simple, and, yeah, "1 + 1" -> "2" is reasonably easy, but in reality there aren't a lot of those super easy changes. Most of the time there is something confounding the situation. Truly neutral refactorings are pretty uncommon in their own right. You can see that when someone is discussing semantic versioning and pointing out that if you define a "major version" as "there exists at least one possible use of the code whose behavior will be changed as a result of this library change", almost any API change is automatically a major version change, which isn't really what anyone wants. E.g., in Python, the mere fact that introspecting on an object's methods will show one more method than it used to isn't really what we want a major version change for. In general, proving refactorings are actually 100% safe is equally difficult; even simple arithmetic changes can result in things overflowing at different times or in different ways, it's virtually impossible to rewrite an expression involving floats without the change being witnessable somehow, extracting a function could make it so that code that previously didn't overflow the stack now does, memory allocation changes can be the difference between OOMing and not and may interact with GC in unpredictable ways if you get really precise, etc.
    - kstrauser 3y ago
      Here's a fun related article on Indistinguishability Obfuscation: https://cacm.acm.org/research/indistinguishability-obfuscation-from-well-founded-assumptions/ https://cacm.acm.org/research/indistinguishability-obfuscati... TL;DR verifying that 2 functions have the same output is really freaking hard.
- Chris_Newton 3y ago
  if you do this your difftool becomes a compiler Some linters and formatters are effectively compilers already, so that doesn’t seem completely implausible in itself. Finding canonical representations of common coding patterns so you can quickly and reliably determine that they are equivalent is a different question, though.
otherjason 3y ago
Difftastic is a useful tool, but in my experience, it's far too slow to be suitable as the default selection for a ubiquitous tool like git.
- drcongo 3y ago
  I'm finding it instantaneous here on a large dirty codebase. In what way is it slow for you?
  - acdha 3y ago
    Diff a large JSON file - I think it has to do with how large a single structure since I notice it most with static test fixtures.
    - drcongo 3y ago
      Interesting, thanks, I'll keep an eye out for that. I rarely need to diff particularly large files.
      - acdha 3y ago
        Yeah, it’s the kind of thing you notice when you enable it as the default git diff helper and it’s great almost all of the time until you hit that one weird repo which has gigantic data files.
kstrauser 3y ago
I think shipping good ol' diff as the default makes sense. It's going to be there already on any system you might want to run git on, it's fast, it's tiny, and everyone knows the basics of how to use it. But I'm glad it's easy to change that default.
rob74 3y ago
> I am curious if there’s been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same) So when using such a diff tool you can spend hours refactoring something, and then git will refuse to commit your changes because your refactoring was successful in not changing the behavior of the code? I understand what you mean, but if we arrive at that point maybe we should stop calling it "diff", to avoid confusion...
- kstrauser 3y ago
  Git doesn't use the output of `diff` to determine whether anything has changed.
  - samatman 3y ago
    True, although not widely known it would seem. It does use diff to generate patches, however. I know in today's GitHub-dominated landscape, that's considered a bit of a dusty feature, but it would be a pity to break it.
    - viraptor 3y ago
      If you want to generate patches rather than look at local differences, there's a specific command for that - https://git-scm.com/docs/git-format-patch https://git-scm.com/docs/git-format-patch
DarkPlayer 3y ago
> I am curious if there's been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same) We are working on https://semanticdiff.com/ https://semanticdiff.com/ which detects basic semantic changes like converting a literal from decimal to hex or reordering keys within JSON objects. It is not a command line utility but a VS Code extension and GitHub App. You can check out https://semanticdiff.com/blog/semanticdiff-vs-difftastic/ https://semanticdiff.com/blog/semanticdiff-vs-difftastic/ if you want to learn more about how it works and how it differs from difftastic.
- Izikiel43 3y ago
  Thank you, you just simplified my life greatly, will use it for a demo tomorrow.
more-coffee 3y ago
I'm trying difft for git now and I also really like it. One reason why I think it shouldn't be the default is that it hides whitespace differences. Maybe that's configurable though, haven't looked into that

ruined 3y ago

>I am curious if there’s been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same). if you do this your difftool becomes a compiler

hobs 3y ago
That's exactly what I have done with diffing SQL in lazy mode - just use a server and diff the AST/plan.
- slotrans 3y ago
  Two semantically equivalent SQL statements can plan differently...
  - rrrrrrrrrrrryan 3y ago
    The exact same SQL statement can plan differently if table statistics change.
  - hobs 3y ago
    Absolutely and for this case a different plan mattered.
mlavrent 3y ago
Sorry, I should've been clearer. I'm interested if there's any tool that does this kind of thing statically, without running the code. I guess a simple approach is to compile both programs and see if the generated code is the same, but I'd guess reasoning at the generated-code level will probably produce a lot more false positives (i.e. tool will report a change when there isn't one) than if you reason about the original program.
- jerf 3y ago
  This gets really hard, really fast. That is, yes, reasonably obviously doing this completely 100% accurately requires a solution to the halting problem, but even getting to "useful" is really really hard. Even the Haskell world doesn't try to solve the "equivalence of functions" problem, and it's even more complicated in imperative languages. You probably have a mental image of catching something really simple, and, yeah, "1 + 1" -> "2" is reasonably easy, but in reality there aren't a lot of those super easy changes. Most of the time there is something confounding the situation. Truly neutral refactorings are pretty uncommon in their own right. You can see that when someone is discussing semantic versioning and pointing out that if you define a "major version" as "there exists at least one possible use of the code whose behavior will be changed as a result of this library change", almost any API change is automatically a major version change, which isn't really what anyone wants. E.g., in Python, the mere fact that introspecting on an object's methods will show one more method than it used to isn't really what we want a major version change for. In general, proving refactorings are actually 100% safe is equally difficult; even simple arithmetic changes can result in things overflowing at different times or in different ways, it's virtually impossible to rewrite an expression involving floats without the change being witnessable somehow, extracting a function could make it so that code that previously didn't overflow the stack now does, memory allocation changes can be the difference between OOMing and not and may interact with GC in unpredictable ways if you get really precise, etc.
  - kstrauser 3y ago
    Here's a fun related article on Indistinguishability Obfuscation: https://cacm.acm.org/research/indistinguishability-obfuscation-from-well-founded-assumptions/ https://cacm.acm.org/research/indistinguishability-obfuscati... TL;DR verifying that 2 functions have the same output is really freaking hard.
Chris_Newton 3y ago
if you do this your difftool becomes a compiler Some linters and formatters are effectively compilers already, so that doesn’t seem completely implausible in itself. Finding canonical representations of common coding patterns so you can quickly and reliably determine that they are equivalent is a different question, though.

otherjason 3y ago

Difftastic is a useful tool, but in my experience, it's far too slow to be suitable as the default selection for a ubiquitous tool like git.

drcongo 3y ago
I'm finding it instantaneous here on a large dirty codebase. In what way is it slow for you?
- acdha 3y ago
  Diff a large JSON file - I think it has to do with how large a single structure since I notice it most with static test fixtures.
  - drcongo 3y ago
    Interesting, thanks, I'll keep an eye out for that. I rarely need to diff particularly large files.
    - acdha 3y ago
      Yeah, it’s the kind of thing you notice when you enable it as the default git diff helper and it’s great almost all of the time until you hit that one weird repo which has gigantic data files.

kstrauser 3y ago

I think shipping good ol' diff as the default makes sense. It's going to be there already on any system you might want to run git on, it's fast, it's tiny, and everyone knows the basics of how to use it. But I'm glad it's easy to change that default.

rob74 3y ago

> I am curious if there’s been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same) So when using such a diff tool you can spend hours refactoring something, and then git will refuse to commit your changes because your refactoring was successful in not changing the behavior of the code? I understand what you mean, but if we arrive at that point maybe we should stop calling it "diff", to avoid confusion...

kstrauser 3y ago
Git doesn't use the output of `diff` to determine whether anything has changed.
- samatman 3y ago
  True, although not widely known it would seem. It does use diff to generate patches, however. I know in today's GitHub-dominated landscape, that's considered a bit of a dusty feature, but it would be a pity to break it.
  - viraptor 3y ago
    If you want to generate patches rather than look at local differences, there's a specific command for that - https://git-scm.com/docs/git-format-patch https://git-scm.com/docs/git-format-patch

DarkPlayer 3y ago

> I am curious if there's been any work on _semantic_ diff tools as well (for when eg the syntax changes but the meaning is the same) We are working on https://semanticdiff.com/ https://semanticdiff.com/ which detects basic semantic changes like converting a literal from decimal to hex or reordering keys within JSON objects. It is not a command line utility but a VS Code extension and GitHub App. You can check out https://semanticdiff.com/blog/semanticdiff-vs-difftastic/ https://semanticdiff.com/blog/semanticdiff-vs-difftastic/ if you want to learn more about how it works and how it differs from difftastic.

Izikiel43 3y ago
Thank you, you just simplified my life greatly, will use it for a demo tomorrow.

more-coffee 3y ago

I'm trying difft for git now and I also really like it. One reason why I think it shouldn't be the default is that it hides whitespace differences. Maybe that's configurable though, haven't looked into that