4 ms·
This is an interesting article, but there's one part that I don't really understand: 'After scaling up the approach by increasing the dataset size, we de-anonym
by Tinyyy 11y ago
This is an interesting article, but there's one part that I don't really understand: 'After scaling up the approach by increasing the dataset size, we de-anonymize 600 programmers with 52% accuracy.'.
Isn't 52% close to a random guess? I don't get how this ties in with the rest of the paragraph either.
- teraflop 11y agoNo, a random guess would give you 0.16% accuracy.
- Lawtonfogle 11y agoIf they were saying that in 52% of the cases, they could guess the correct choice out of 600, then random chance alone would have been <.2%. If they were saying that in 52% of cases, they could guess if some code did or did not belong to a given programmer, then it is basically a random guess.
- nothrabannosir 11y ago> If they were saying that in 52% of cases, they could guess if some code did or did not belong to a given programmer, then it is basically a random guess. That entirely depends on the distribution of the original sample.
- dllthomas 11y agoDoes it? If you're asked "does this code belong to this programmer?" and flip a coin, you'll approximate 50% regardless of distribution of programmers. Of course, "const No" might do quite a bit better than 50%, depending on the distribution of the original sample (for that matter, so might "const Yes", but those distributions seem less likely).
- michaelt 11y agoWell, if there were 600 programmers, one of whom had written 649 programs and 599 of whom had written 1 program each, you could achieve 52% accuracy by always guessing the same guy. Of course, the paper explicitly says "For this experiment, we use 600 contestants from [Google Code Jam] with 9 files" so I think in this case the distribution was probably fairly even?
- dllthomas 11y agoI think you missed some context. Up-thread, someone raised the point that it's unclear whether the question was "Of these 600 programmers, which wrote this code?" or "Did this programmer write this code?", and this subthread is discussing the latter case.
- Lawtonfogle 11y agoYeah, I had an underlying assumption in my mental model that they would have split the questions so that half were yes and half were no. But you are right that that assumption may not hold. A random programmer with a random program will be a 'no' far more than 50% of the time.
- mrdrozdov 11y agoConsidering there are 600 programmers, 0.166% (1/600) would be random guess.
- kazinator 11y ago52% is close to a random guess, yes: between two alternative outcomes, such as heads or tails in a coin toss. If the question is, which of these 20 programmers wrote this code", then a random guess has only a 5% chance (1/20) of being right! So 52% is more than 10X better than random.