6 ms·
Here is what CoPilot wrote in Python: # A function to swap the sixth bit and seventeenth bit of a 32-bit signed integer. def swap_bits(x): # Get the
by quenix 5y ago
Here is what CoPilot wrote in Python:
# A function to swap the sixth bit and seventeenth bit of a 32-bit signed integer.
def swap_bits(x):
# Get the value of the sixth bit.
bit6 = x & (1 << 5)
# Get the value of the seventeenth bit.
bit17 = x & (1 << 16)
# Swap the values of the sixth and seventeenth bit.
bit6 = bit6 >> 5
bit17 = bit17 >> 16
# Combine the swapped values of the sixth and seventeenth bit.
x = x ^ (bit6 << 16)
x = x ^ (bit17 << 5)
return x
It actually synthesised about 10 viable solutions. Some interesting ones:
def swap_six_seventeen(x):
# Get the binary representation of the integer.
binary = bin(x)[2:]
# Add zeros to the beginning of the binary representation.
binary = '0' * (32 - len(binary)) + binary
# Swap the sixth and seventeenth bit.
binary = binary[:5] + binary[17] + binary[5:17] + binary[18:]
# Convert the binary back to an integer.
return int(binary, 2)
- 37ef_ced3 5y agoAll wrong. These "solutions" are not correct.
- UncleMeat 5y agoThe xors are wrong. Stick in the integer 1<<16.
- omnicognate 5y agoThe first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg. bin(swap_bits(0b_1_0000000000_0_00000)) '0b10000000000100000' bin(swap_bits(0b_0_0000000000_1_00000)) '0b10000000000100000' bin(swap_bits(0b_1_0000000000_1_00000)) '0b0' bin(swap_bits(0b_0_0000000000_0_00000)) '0b0' The second one converts the value to a string and uses string operations, which is wildly inefficient and a very common mistake made by inexperienced programmers unaware of bitwise operations (so presumably common in the training set). It also attempts to swap the 6th and 17th most significant bits rather than the 6th and 17th least significant bits, i.e. counts in the opposite direction to the first one (the comment doesn't specify but typically you count from the least significant bit in these situations). Worse, though, it gets the string manipulation completely wrong. I think it's trying for `binary[:5] + binary[16] + binary[6:16] + binary[5] + binary[17:]`, i.e. characters 1-5, then character 17, then characters 7-16, then character 6, then characters 18-32. The manipulation it does just completely mangles the string. I'm very keen to try Github Copilot if they ever admit me to the beta (I've been waiting forever) and will adopt it enthusiastically if it's useful. However, this is exactly what I've pessimistically expected. Analysing these truly awful implementations to identify the subtle and bizarre misbehaviours has taken me far, far longer than it would have taken me to just write and test a working implementation myself. And I'm supposed to evaluate 10 of these to see if one of them might possibly do the right thing?!?!
- Veedrac 5y agoThe first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.
- 37ef_ced3 5y agoIt illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion. Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.
- Veedrac 5y agoIt got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.
- 37ef_ced3 5y agoIf you ask CoPilot to solve something it hasn't seen, it won't be able to solve it. It's a transformer. Do you understand what that means? It's just matrix multiplication. It generates maximum likelihood token strings, based on its training data. It doesn't "understand" what those token string mean. You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training. Maximum likelihood token strings. Period.