3 ms·
an interesting question is how to filter out the folks who just feed it through chatgpt, because obviously the people who just lie to get out of any mild annoya
by kragen 3y ago
an interesting question is how to filter out the folks who just feed it through chatgpt, because obviously the people who just lie to get out of any mild annoyance are the ones you least want as coworkers (though obviously people who can effectively use large language models will be incredibly valuable)
i asked gpt-4 some programming questions today, and although it gave clear and convincing explanations in english, it made a lot of really dumb mistakes in writing the code. but this was in a mode where it couldn't call out to an external interpreter; maybe a lot of those will go away once it can test the code
most programmers would have done worse, but is there a pattern to the dumb mistakes chatgpt makes that is clearly different from the dumb mistakes junior programmers make?
like for example i gave it this
#!/usr/bin/perl
# See also cifs.py. Edited GPT-4 output; see <https://bin.gy/clignitter>
# Try watch -n .1 ./wing.pl.
for(1..9999){$t=0|rand 3;($y,$x)=$t?($y/2,$x/2+($t-1?40:10)):(.3*$x-.4*$y+12,
.4*$x+(.2+abs($$%99/99-.5))*$y);$a[$x][$y]=1}map{$r=$_*2;print map({substr
" .':",2*$_->[$r]+$_->[$r+1],1}@a[0..79]),"\n"}(0..18);
and asked it to translate it to lua, getting the following nonsense in response
math.randomseed(os.time())
local a = {}
for i=1,9999 do
local t = math.floor(math.random() * 3)
local x,y
if t == 0 then
x,y = .3*a[1][2]-.4*a[2][1]+12,
.4*a[1][2]+(.2+math.abs(os.time()%99/99-.5))*a[2][1]
else
x,y = a[1][2]/2, a[2][1]/2+(t-1 and 40 or 10)
end
if not a[x] then a[x] = {} end
a[x][y] = true
end
local str = " .':"
for r=0,18 do
for c=0,79 do
local val = (a[c*2] and (a[c*2][r*2] or 0) or 0) + (a[c*2+1] and
(a[c*2+1][r*2+1] or 0) or 0)
io.write(string.sub(str,val,val))
end
print()
end
of course, i've seen worse from interviewees, and understanding the original program is a high (and arguably counterproductive) bar, but perhaps the particular pattern of bonehead errors here could identify this as llm output? maybe the very fact of being so spectacular at understanding the original code is a red flag, at least combined with being so incompetent at writing the lua version?
- Our_Benefactors 3y ago> because obviously the people who just lie to get out of any mild annoyance are the ones you least want as coworkers (though obviously people who can effectively use large language models will be incredibly valuable) So people who can use them effectively, but not like that! Do you not see the irony in this statement?
- hluska 3y agoTheir verb was ‘lie’. Do you want a lying coworker?
- Our_Benefactors 3y agoHave you stopped beating your wife yet? I don’t see using an LLM as a coding assistant as “lying” anyways.
- kragen 3y agousing an llm as a coding assistant is not lying in general, just in this context, where you're taking a 'skill assessment test', that is, being quizzed to find out what you know how to do
- hluska 3y agoOne intellectually void comment and now this offensive one. You’re a waste of time.
- Our_Benefactors 3y agoI disagree with the premise of your question entirely because using LLMs is not lying.
- hluska 3y agoThen develop social skills and say that. This shit about asking me about beating my wife is very offensive and there was absolutely no point. You’re just being pointlessly offensive because someone hurt your ego.
- arp242 3y agoIf you say "I programmed this myself" or "I did not use LLMs" while in fact you did then you misrepresented reality. That is, you lied. It baffles me this is even a point of contention.
- RagnarD 3y agoThis underscores the problem with GPT-4 currently. It apparently can't do what any legitimate programmer can do: look at code and simulate operations in their head. Of course programs need to be run on actual hardware with real program support to find bugs, but they have to first be written and confirmed correct as well as possible, mentally. As it stands now they're doing some kind of fuzzy pattern matching that's kinda-sorta plausible and often completely wrong on inspection. And amusingly (?) if you ask it to try again, it'll come up quite different and also completely wrong code. To be really good at programming these systems need the equivalent of the "understanding" a human programmer has, not just some abstract fuzzy pattern matching.
- kragen 3y agomy impression is that it is a hell of a lot better than i am at simulating operations in its head, but not nearly as good as a regular interpreter; it's at least enormously faster at coming up with mostly correct explanations of tricky code
- kragen 3y agoon fully debugging the above i'm no longer sure most programmers would do worse; by my count i fixed 11 bugs in 21 lines of code. http://canonical.org/~kragen/sw/dev3/wing.lua http://canonical.org/~kragen/sw/dev3/wing.lua has all the gruesome details