7 ms·
The example problem (essentially, is T a subsequence of S with deletions of size N) is a classic problem with no doubt dozens of implementations in AlphaCode's
by 37ef_ced3 5y ago
The example problem (essentially, is T a subsequence of S with deletions of size N) is a classic problem with no doubt dozens of implementations in AlphaCode's training set.
And yet, what a garbage solution it produces.
To illustrate the difference between intelligence and regurgitation, someone tell me what CoPilot generates for this:
// A Go function to swap the sixth bit and seventeenth bit of a 32-bit signed integer.
Here is a human solution:
func swap(x int32) int32 {
const mask = 1 << 5
var (
xor1 = (x>>11 ^ x) & mask
xor2 = xor1 << 11
)
return x ^ xor1 ^ xor2
}
CoPilot cannot reason numerically like this (understand "seventeenth bit" and "sixth bit" and generate the right code for that combination). It needs to understand the size of the gap between the bits, i.e., 11, and that's too hard.
- quenix 5y agoHere is what CoPilot wrote in Python: # A function to swap the sixth bit and seventeenth bit of a 32-bit signed integer. def swap_bits(x): # Get the value of the sixth bit. bit6 = x & (1 << 5) # Get the value of the seventeenth bit. bit17 = x & (1 << 16) # Swap the values of the sixth and seventeenth bit. bit6 = bit6 >> 5 bit17 = bit17 >> 16 # Combine the swapped values of the sixth and seventeenth bit. x = x ^ (bit6 << 16) x = x ^ (bit17 << 5) return x It actually synthesised about 10 viable solutions. Some interesting ones: def swap_six_seventeen(x): # Get the binary representation of the integer. binary = bin(x)[2:] # Add zeros to the beginning of the binary representation. binary = '0' * (32 - len(binary)) + binary # Swap the sixth and seventeenth bit. binary = binary[:5] + binary[17] + binary[5:17] + binary[18:] # Convert the binary back to an integer. return int(binary, 2)
- 37ef_ced3 5y agoAll wrong. These "solutions" are not correct.
- UncleMeat 5y agoThe xors are wrong. Stick in the integer 1<<16.
- omnicognate 5y agoThe first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg. bin(swap_bits(0b_1_0000000000_0_00000)) '0b10000000000100000' bin(swap_bits(0b_0_0000000000_1_00000)) '0b10000000000100000' bin(swap_bits(0b_1_0000000000_1_00000)) '0b0' bin(swap_bits(0b_0_0000000000_0_00000)) '0b0' The second one converts the value to a string and uses string operations, which is wildly inefficient and a very common mistake made by inexperienced programmers unaware of bitwise operations (so presumably common in the training set). It also attempts to swap the 6th and 17th most significant bits rather than the 6th and 17th least significant bits, i.e. counts in the opposite direction to the first one (the comment doesn't specify but typically you count from the least significant bit in these situations). Worse, though, it gets the string manipulation completely wrong. I think it's trying for `binary[:5] + binary[16] + binary[6:16] + binary[5] + binary[17:]`, i.e. characters 1-5, then character 17, then characters 7-16, then character 6, then characters 18-32. The manipulation it does just completely mangles the string. I'm very keen to try Github Copilot if they ever admit me to the beta (I've been waiting forever) and will adopt it enthusiastically if it's useful. However, this is exactly what I've pessimistically expected. Analysing these truly awful implementations to identify the subtle and bizarre misbehaviours has taken me far, far longer than it would have taken me to just write and test a working implementation myself. And I'm supposed to evaluate 10 of these to see if one of them might possibly do the right thing?!?!
- Veedrac 5y agoThe first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.
- 37ef_ced3 5y agoIt illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion. Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.
- c4wrd 5y agoAsk and you shall receive, here is the output it gave me: // Write a Go function to swap the sixth bit and seventeenth bit of a 32-bit signed integer. package main import ( "fmt" "math" ) func main() { var a int32 = 0b1010101010101010101010101010101010101010101010101010101010101010 fmt.Printf("%b\n", a) fmt.Printf("%b\n", swapBits(a, 6, 17)) } func swapBits(a int32, i int, j int) int32 { // convert to binary bin := fmt.Sprintf("%b", a) // get the bits bit1 := bin[i-1 : i] bit2 := bin[j-1 : j] // swap the bits bin = bin[:i-1] + bit2 + bin[i:] bin = bin[:j-1] + bit1 + bin[j:] // convert back to int return int32(bin2int(bin)) } func bin2int(bin string) int64 { var sum int64 for i, v := range bin { if v == '1' { sum += int64(math.Pow(2, float64(len(bin)-i-1))) } } return sum }
- 37ef_ced3 5y agoRidiculous. It's a joke, right? Perhaps the most inefficient and naive solution ever? CoPilot is regurgitating some solution from its training set, the solution of an inept programmer who would manipulate bits via conversion to string... yikes.
- skulk 5y agoThe next iteration of code assistant needs to be able to parse responses like your comment and update the code accordingly. Once a human+computer pair can converge on a correct and admissible solution to _any_ tractable programming task through natural language dialogue, we should start worrying about our jobs going away. Until then, for each line of code generated by AI, there will be two jobs created to maintain that code.
- hackinthebochs 5y agoWhich direction in feature space do you move in response to "you inept POS"?
- electroly 5y agoCopilot can do that, sorta. You undo the completion and add something like "... but don't convert it to a string" to the comment, then have it try completing again.
- deleted 5y ago[deleted]
- altcognito 5y agoWould we be able to generate unit tests? Strikes me that this would be important to verify given that we didn't even "write" the code. At some point we might not even be looking at the generated code? I almost guarantee that's what is going to happen eventually.
- 37ef_ced3 5y agoYou can see it happening already. Solutions are posted, and they're wrong. But the CoPilot user can't see the code is wrong.
- dskloet 5y agoThere's really no need for an 11 in the code. I'd say that makes the code worse, not better.
- 37ef_ced3 5y agoThis is a toy problem to illustrate that CoPilot cannot write code that requires mathematical reasoning. It regurgitates solutions from the training set, via a mixed internal reresentation.
- dskloet 5y agoWhat requires mathematical reasoning? Getting or setting the nth bit? Or swapping two variables? What am I missing?
- deanmen 5y agounsigned int swapbits(unsigned int a) { bool bit6 = a & (1 << 5); bool bit17 = a & (1 << 16); if (bit6 == bit17) return a; //bits are the same, do nothing return (a ^ (1 << 5) ^ (1 << 16)); // flip both 6th and 17th bits }
- errcorrectcode 5y agoGross and not portable C99. #define B6 (1<<5) #define B17 (1<<16) unsigned swapbits(unsigned a) { return ((a & B6 == a & B17) ? a : (a ^ (B6 | B17))); } Here's some BFP: unsigned swapbits(unsigned a) { unsigned flip = (a & B6 == a & B17); return (a ^ ((flip<<5) | (flip<<16))); } int and double are C's implicit lingua francas for underspecified literals and implicit type conversions. Throwing int everywhere is redundant like "ATM machine."
- deanmen 5y agoThe definition of flip requires parenthesis (a & B6) == (a & B17) as == has higher precedence than and. int is required in C++ but not in C as you said.
- deleted 5y ago[deleted]
- deanmen 5y agoYou can do it without a subtraction unsigned int swapbits(unsigned int a) { bool bit6 = a & (1 << 5); bool bit17 = a & (1 << 16); if (bit6 == bit17) return a; //bits are the same, do nothing return (a ^ (1 << 5) ^ (1 << 16)); // flip both 6th and 17th bits }
- 37ef_ced3 5y agoAnd, to be clear, this is a human solution. Not as efficient as mine, but kudos.
- deanmen 5y agoThe compiler seems to generate less efficient code than either if you write the most mechanical solution for swapping the bits in C. gcc and clang give swap: # @swap mov ecx, edi shr ecx, 11 and ecx, 32 mov eax, edi and eax, -65569 or eax, ecx and edi, 32 shl edi, 11 or eax, edi ret swap: mov eax, edi mov edx, edi and edi, -65569 sal eax, 11 shr edx, 11 and eax, 65536 and edx, 32 or eax, edx or eax, edi ret /* only works on little-endian! */ typedef union { struct { unsigned bit1: 1; unsigned bit2: 1; unsigned bit3: 1; unsigned bit4: 1; unsigned bit5: 1; unsigned bit6: 1; unsigned bit7: 1; unsigned bit8: 1; unsigned bit9: 1; unsigned bit10: 1; unsigned bit11: 1; unsigned bit12: 1; unsigned bit13: 1; unsigned bit14: 1; unsigned bit15: 1; unsigned bit16: 1; unsigned bit17: 1; unsigned bit18: 1; unsigned bit19: 1; unsigned bit20: 1; unsigned bit21: 1; unsigned bit22: 1; unsigned bit23: 1; unsigned bit24: 1; unsigned bit25: 1; unsigned bit26: 1; unsigned bit27: 1; unsigned bit28: 1; unsigned bit29: 1; unsigned bit30: 1; unsigned bit31: 1; unsigned bit32: 1; }; unsigned int n; } mybits; unsigned int swap(unsigned int n) { mybits foo; foo.n = n; unsigned tmp = foo.bit6; foo.bit6 = foo.bit17; foo.bit17 = tmp; return foo.n; }