5 ms·

I actually think this feature would be fine if implemented better. GPT-4 is perfectly capable of accurately explaining what a short code block does if you promp

by thorum 3y ago

I actually think this feature would be fine if implemented better. GPT-4 is perfectly capable of accurately explaining what a short code block does if you prompt it with relevant documentation… which MDN has in abundance, but isn’t using, for some reason. The mistakes seem to be (1) using GPT 3.5 instead because it’s cost effective, even though GPT 3.5 has severe issues with hallucination, and (2) not including context from the surrounding article into the prompt. Also, since I assume MDN has a finite number of things that need explaining, why not let users mark an AI explanation as wrong, and submit their own corrected versions?

contravariant 3y ago
> Also, since I assume MDN has a finite number of things that need explaining, why not let users mark an AI explanation as wrong, and submit their own corrected versions? I mean if you implement it well you don't really need the model. If anything it would be more effective if it was always just slightly wrong.
sadgfasfgiono 3y ago
I can't imagine any situation where it would be acceptable. I have personally wasted hundreds of hours over the course of my career due to incorrect documentation. There are millions of programmers in the world. Incorrect documentation has surely caused hundreds of millions of hours of wasted labor, or tens of billions of dollars of economic damage. That's just the damage of wasted development time, not the harm buggy software causes users.
acdha 3y ago
For me, the most obvious problem is that it’s often mixing wrong information in. I found a couple of very simple examples where it accurately explained things but anything longer tended to have multiple incorrect points On a deeper level, though, I don’t know that it’s actually useful. I haven’t found a single example where it added anything which wasn’t present in the authored text, and the explanations tend to be very pointless mechanical descriptions which miss the larger point of why you’d want to do things. On the popover API page, for example, it had a bunch of regurgitated comments before telling me that you’d need to write a bunch of JavaScript to actually toggle popover visibility when the whole point of that API is not needing to do that. That was also interesting with the incorrect claim that it was a non-standard attribute and you’d need to implement it using a JavaScript framework like Vue.js when you think about how much code someone might write if they relied on that explanation.
justinclift 3y ago
Does GPT-4 really never hallucinate? This is the first time I've personally heard someone claiming anything like that, though I don't tend to do anything with LLMs (due to this bullshit factor).
- dns_snek 3y ago
  Anyone who says that GPT-4 doesn't hallucinate is either clueless or an AI grifter, though I don't think GP was claiming that, just calling out that GPT-3.5 is really, really bad at it.
  - justinclift 3y ago
    Cool, thanks for clarifying. Was hoping they'd managed to crack that nut, but doesn't seem like it yet. :)

by thorum 3y ago

contravariant 3y ago

> Also, since I assume MDN has a finite number of things that need explaining, why not let users mark an AI explanation as wrong, and submit their own corrected versions? I mean if you implement it well you don't really need the model. If anything it would be more effective if it was always just slightly wrong.

sadgfasfgiono 3y ago

I can't imagine any situation where it would be acceptable. I have personally wasted hundreds of hours over the course of my career due to incorrect documentation. There are millions of programmers in the world. Incorrect documentation has surely caused hundreds of millions of hours of wasted labor, or tens of billions of dollars of economic damage. That's just the damage of wasted development time, not the harm buggy software causes users.

acdha 3y ago

For me, the most obvious problem is that it’s often mixing wrong information in. I found a couple of very simple examples where it accurately explained things but anything longer tended to have multiple incorrect points On a deeper level, though, I don’t know that it’s actually useful. I haven’t found a single example where it added anything which wasn’t present in the authored text, and the explanations tend to be very pointless mechanical descriptions which miss the larger point of why you’d want to do things. On the popover API page, for example, it had a bunch of regurgitated comments before telling me that you’d need to write a bunch of JavaScript to actually toggle popover visibility when the whole point of that API is not needing to do that. That was also interesting with the incorrect claim that it was a non-standard attribute and you’d need to implement it using a JavaScript framework like Vue.js when you think about how much code someone might write if they relied on that explanation.

justinclift 3y ago

Does GPT-4 really never hallucinate? This is the first time I've personally heard someone claiming anything like that, though I don't tend to do anything with LLMs (due to this bullshit factor).

dns_snek 3y ago
Anyone who says that GPT-4 doesn't hallucinate is either clueless or an AI grifter, though I don't think GP was claiming that, just calling out that GPT-3.5 is really, really bad at it.
- justinclift 3y ago
  Cool, thanks for clarifying. Was hoping they'd managed to crack that nut, but doesn't seem like it yet. :)