5 ms·

At my first professional programming gig the company I worked for was selling items on Amazon. The listings were big ticket items that had several sellers compe

by Timpy 7y ago

At my first professional programming gig the company I worked for was selling items on Amazon. The listings were big ticket items that had several sellers competing on them. Any seller could change some of the details of the listing, so suddenly all their competition would be shipping the item with wrong specs. I wrote a program that periodically grabbed the titles of our live Amazon listings and compared them to our database of what we actually wanted the title to read. I used Levenshtein Distance as a sort of "severity of change" metric, the program would sort the changes accordingly and send an email to a person on the sales team. It was fun implementing Levenshtein Distance, but it wasn't a perfect metric for this use case. Some of the most severe changes would be a single digit changing; e.g. "2gb" vs "8gb"

LorenPechtel 7y ago
You can modify the basic algorithm to weight changes differently. I had the opposite case--differing numbers were more likely to be what the user was looking for. I made digit changes cost 1 point, one string being longer cost 1 point per character (suffixes were another case where it was likely the objective) and everything else cost 2.
- nwallin 7y ago
  Someday I want to write a diff utility specifically for code. Whitespace addition/removal cost like .1, addition/removal of a curly brace .5. There are a handful of other things default diff does that make it really annoying for code.
  - apjr 7y ago
    I did this for a school project, because students would "cheat" on group projects by exclusively changing whitespace in diffs, or moving code around, thereby making it look like they had large commits. We tracked code "moves", whitespace changes, and "trivial" changes via Levenshtein distance--which isn't great for capturing most simple renamings--and then marked them up separately in an "augmented diff". We briefly looked into doing AST edit-distance, but that turned out to be infeasible for many reasons. Over-all it helped the prof grade faster, at least in the degenerate cases.
    - bonoboTP 7y ago
      Measuring a student's impact by a project on the number of lines of code they committed can be very misleading. Sure, in the extremes it will give sensible results (someone wrote only 3 lines, or wrote 90% of the code base). But minor, questionably motivated refactorings can create huge commits, while someone may come up with some great design while walking around for hours and sketching on paper, which all ultimately boils down to a short function. Dumb metrics like this cannot replace actually talking with the students, asking questions about their choices and alternatives, having them do presentations or lab notebooks/logbooks etc.
      - Cogito 7y ago
        > Dumb metrics like this cannot replace actually talking with the students, asking questions about their choices and alternatives, having them do presentations or lab notebooks/logbooks etc. As said in the parent: >> Over-all it helped the prof grade faster, at least in the degenerate cases.
  - truncate 7y ago
    I never actually used any of these, but have stumbled upon some tools which create/apply diffs on ASTs of your source rather than lines[1]. I think the keywords to search is "semantic diff/patch/merge", "AST diff" or some combination of these. [1] https://lwn.net/Articles/315686/ https://lwn.net/Articles/315686/
  - 9nGQluzmnq3M 7y ago
    Diff already has a number of options for ignoring whitespace changes: https://www.gnu.org/software/diffutils/manual/html_node/White-Space.html#White-Space https://www.gnu.org/software/diffutils/manual/html_node/Whit...
galkk 7y ago
Same thing: "Math, 4 grade" and "Math, 5 grade" are very different books, even if they are by the same author :)

by Timpy 7y ago

LorenPechtel 7y ago

You can modify the basic algorithm to weight changes differently. I had the opposite case--differing numbers were more likely to be what the user was looking for. I made digit changes cost 1 point, one string being longer cost 1 point per character (suffixes were another case where it was likely the objective) and everything else cost 2.

nwallin 7y ago
Someday I want to write a diff utility specifically for code. Whitespace addition/removal cost like .1, addition/removal of a curly brace .5. There are a handful of other things default diff does that make it really annoying for code.
- apjr 7y ago
  I did this for a school project, because students would "cheat" on group projects by exclusively changing whitespace in diffs, or moving code around, thereby making it look like they had large commits. We tracked code "moves", whitespace changes, and "trivial" changes via Levenshtein distance--which isn't great for capturing most simple renamings--and then marked them up separately in an "augmented diff". We briefly looked into doing AST edit-distance, but that turned out to be infeasible for many reasons. Over-all it helped the prof grade faster, at least in the degenerate cases.
  - bonoboTP 7y ago
    Measuring a student's impact by a project on the number of lines of code they committed can be very misleading. Sure, in the extremes it will give sensible results (someone wrote only 3 lines, or wrote 90% of the code base). But minor, questionably motivated refactorings can create huge commits, while someone may come up with some great design while walking around for hours and sketching on paper, which all ultimately boils down to a short function. Dumb metrics like this cannot replace actually talking with the students, asking questions about their choices and alternatives, having them do presentations or lab notebooks/logbooks etc.
    - Cogito 7y ago
      > Dumb metrics like this cannot replace actually talking with the students, asking questions about their choices and alternatives, having them do presentations or lab notebooks/logbooks etc. As said in the parent: >> Over-all it helped the prof grade faster, at least in the degenerate cases.
- truncate 7y ago
  I never actually used any of these, but have stumbled upon some tools which create/apply diffs on ASTs of your source rather than lines[1]. I think the keywords to search is "semantic diff/patch/merge", "AST diff" or some combination of these. [1] https://lwn.net/Articles/315686/ https://lwn.net/Articles/315686/
- 9nGQluzmnq3M 7y ago
  Diff already has a number of options for ignoring whitespace changes: https://www.gnu.org/software/diffutils/manual/html_node/White-Space.html#White-Space https://www.gnu.org/software/diffutils/manual/html_node/Whit...

galkk 7y ago

Same thing: "Math, 4 grade" and "Math, 5 grade" are very different books, even if they are by the same author :)