3 ms·

I used to agree, but as an experienced engineer asking about rust and y-crdt, it sent me down so many wrong rabbit holes with half valid information. I used Cl

by dartos 2y ago

I used to agree, but as an experienced engineer asking about rust and y-crdt, it sent me down so many wrong rabbit holes with half valid information. I used Claude recently to refresh my knowledge on the browser history API and it said that it gets cleared when the user navigates to a new page because the “JavaScript context has changed” I have the experience and know how to verify this stuff, but a new engineer may not, and that would be awful. Things like these made me cancel all my AI subscriptions and just wait for whatever comes after transformers.

cvwright 2y ago
I think of AI like a very eager and bright 20 yr old intern who’s really into whatever tech I’m trying to learn. Good knowledge of facts, willing to give anything a try, but not much wisdom.
- dartos 2y ago
  > Good knowledge of facts I haven’t seen this demonstrated in gpt-4 or Claude sonnet when asking anything beyond the most extreme basics. I consistently get subtly wrong answers and whenever I ask “oh okay, so it works like this” I always get “Yes! Exactly. You show a deep understanding of…” even though I was wrong based on the wrong info from the LLM. Useless for knowledge work beyond RAG, it seems. Search engines that I need to double check are worse than documentation. It’s why so many of us moved beyond stack overflow. Documentation has gotten so good.
- tmpz22 2y ago
  ChatGPT and Claude outputs are equivalent to a 20 yr old intern who goes to MIT?
pizza 2y ago
But actually whenever this happens you get rich signal about the hidden didactic assumptions of your method of prompting it about things you yourself are unsure you can verify for yourself, and also of how you thought the tool worked. This is a good meta skill to hone.
- dartos 2y ago
  In the same way going to a fortune teller can give me rich insight about my love life and work life balance, sure.
  - pizza 2y ago
    A visitor to physicist Niels Bohr's country cottage, noticing a horseshoe hanging on the wall, teased the eminent scientist about this ancient superstition. "Can it be true that you, of all people, believe it will bring you luck?" "Of course not," replied Bohr, "but I understand it brings you luck whether you believe it or not." s/luck/lower negative log likelihood. Besides, a participant can still think about and glean truths from their reflections about a conversation they had which contained only false statements.
    - dartos 2y ago
      What point are you trying to make? That it’s better to be superstitious than not, because maybe it matters?
      - pizza 2y ago
        The internal mechanisms by which a model achieves a low negative log likelihood become irrelevant as it approaches perfect simulation of the true data distribution.
    - memhole 2y ago
      This is how I think about it as well. It’s like the equivalent of trying to build an ML model with only positive examples otherwise
Ferret7446 2y ago
> I have the experience and know how to verify this stuff That was GP's point. Steve Yegge has also talked about this.
- dartos 2y ago
  Is that not the opposite point? Someone who does not have that experience would be much more harmed by the false information LLMs can and do spit out.

by dartos 2y ago

cvwright 2y ago

I think of AI like a very eager and bright 20 yr old intern who’s really into whatever tech I’m trying to learn. Good knowledge of facts, willing to give anything a try, but not much wisdom.

dartos 2y ago
> Good knowledge of facts I haven’t seen this demonstrated in gpt-4 or Claude sonnet when asking anything beyond the most extreme basics. I consistently get subtly wrong answers and whenever I ask “oh okay, so it works like this” I always get “Yes! Exactly. You show a deep understanding of…” even though I was wrong based on the wrong info from the LLM. Useless for knowledge work beyond RAG, it seems. Search engines that I need to double check are worse than documentation. It’s why so many of us moved beyond stack overflow. Documentation has gotten so good.
tmpz22 2y ago
ChatGPT and Claude outputs are equivalent to a 20 yr old intern who goes to MIT?

pizza 2y ago

But actually whenever this happens you get rich signal about the hidden didactic assumptions of your method of prompting it about things you yourself are unsure you can verify for yourself, and also of how you thought the tool worked. This is a good meta skill to hone.

dartos 2y ago
In the same way going to a fortune teller can give me rich insight about my love life and work life balance, sure.
- pizza 2y ago
  A visitor to physicist Niels Bohr's country cottage, noticing a horseshoe hanging on the wall, teased the eminent scientist about this ancient superstition. "Can it be true that you, of all people, believe it will bring you luck?" "Of course not," replied Bohr, "but I understand it brings you luck whether you believe it or not." s/luck/lower negative log likelihood. Besides, a participant can still think about and glean truths from their reflections about a conversation they had which contained only false statements.
  - dartos 2y ago
    What point are you trying to make? That it’s better to be superstitious than not, because maybe it matters?
    - pizza 2y ago
      The internal mechanisms by which a model achieves a low negative log likelihood become irrelevant as it approaches perfect simulation of the true data distribution.
  - memhole 2y ago
    This is how I think about it as well. It’s like the equivalent of trying to build an ML model with only positive examples otherwise

Ferret7446 2y ago

> I have the experience and know how to verify this stuff That was GP's point. Steve Yegge has also talked about this.

dartos 2y ago
Is that not the opposite point? Someone who does not have that experience would be much more harmed by the false information LLMs can and do spit out.