5 ms·

This mostly matches my experience but with one important caveat around using them to learn new subjects. When I'm diving into a wholly new subject for the firs

This mostly matches my experience but with one important caveat around using them to learn new subjects. When I'm diving into a wholly new subject for the first time, in a field totally unrelated to my field (similar to the author, C programming and security) for example biochemistry or philosophy or any field where I don't have even a basic grounding, I still worry about having subtly-wrong ideas about fundamentals being planted early-on in my learning. As a programmer I can immediately spot "is this code doing what I asked it to do" but there's no equivalent way to ask "is this introductory framing of an entire field / problem space the way an actual expert would frame it for a beginner" etc. At the end of the day we've just made the reddit hivemind more eloquent. There's clearly tons of value there but IMHO we still need to be cognizant of the places where bad info can be subtly damaging.

simonw 2y ago
I don't worry about that much at all, because my experience of learning is that you inevitably have to reconsider the fundamentals pretty often as you go along. High school science is a great example: once you get to university you have to un-learn all sorts of things that you learned earlier because they were simplifications that no longer apply. Terry Pratchett has a great quote about this: https://simonwillison.net/2024/Jul/1/terry-pratchett/ https://simonwillison.net/2024/Jul/1/terry-pratchett/ For fields that I'm completely new to, the thing I need most is a grounding in the rough shape and jargon of the field. LLMs are fantastic at that - it's then up to me to take that grounding and those jargon terms and start building my own accurate-as-possible mental model of how that field actually works. If you treat LLMs as just one unreliable source of information (like your well-read friend who's great at explaining things in terms that you understand but may not actually be a world expert on a subject) you can avoid many of the pitfalls. Where things go wrong is if you assume LLMs are a source of irrefutable knowledge.
- lolinder 2y ago
  > like your well-read friend who's great at explaining things in terms that you understand but may not actually be a world expert on a subject I guess part of my problem with using them this way is that I am that well-read friend. I know how the sausage is made, how easy it is to bluff a response to any given question, and for myself I tend to prefer reading original sources to ensure that the understanding that I'm conveying is as accurate as I can make it and not a third-hand account whose ultimate source is a dubious Reddit thread. > High school science is a great example: once you get to university you have to un-learn all sorts of things that you learned earlier because they were simplifications that no longer apply. The difference between this and a bad mental model generated by an LLM is that the high school science models were designed to be good didactic tools and to be useful abstractions in their own right. An LLM output may be neither of those.
  - simonw 2y ago
    If you "tend to prefer reading original sources" then I think you're the best possible candidate for LLM-assisted learning, because you'll naturally use them as a starting point, not the destination. I like to use LLMs to get myself the grounding I need to then start reading further around a topic from more reliable sources. That's a great point about high school models being deliberately designed as didactic tools. LLMs will tend to spit those out too, purely because the high school version of anything has been represented heavily enough in the training data that it's more likely than not to fall out of the huge matrix of numbers!
    - lolinder 2y ago
      > LLMs will tend to spit those out too, purely because the high school version of anything has been represented heavily enough in the training data that it's more likely than not to fall out of the huge matrix of numbers! That assumes that the high school version of the subject exists, which is unlikely because I already have the high school version of most subjects that have a high school version. The subjects that I would want to dig into at that level would be something along the lines of chemical engineering, civil engineering, or economics—subjects that I don't yet know very much about but have interest or utility for me. These subjects don't have a widely-taught high school version crafted by humans, and I don't trust that they would have enough training data to produce useful results from an LLM.
      - fragmede 2y ago
        at what point does a well-read high school-level LLM graduate to college? I asked one about Reinforcement Learning, and at first it treated me like the high schooler, but I was able to prod it into giving me answers more suitable for my level. Of course, I don't know what's hallucinated or not, but it satisfied my curiosity enough to be worth my while. I'm not looking to change careers, so getting things 100% right about in the fields of chemical engineering, civil engineering, or economics isn't necessary. I look at it as the same way I think of astrophysics. After reading Steven Hawkins book, I still don't really know astrophysics at all, but I have a good enough model of things. And as they say, all models are wrong, some are useful. If I were a lawyer using these things for work, I'd be insane to trust one at this stage, but the reality is I'm not using my digging into things I don't know about for anything load bearing, but even if I were, I'd still use an LLM to get started. Eg the post didn't state how the author learned anything the name for the dropped letter O, but I can describe a thing and have the LLM give me the name of it. there's an emphasis on getting things totally 100% right does erode trust, but you get a sense for what's could be a hallucination and then check background resources if you get enough experience with the tool.
safetytrick 2y ago
In the article the author mentions wanting to benchmark a GPU and using ChatGPT to write CUDA. Benchmarks are easy to mess up and to interpret incorrectly without understanding. I see this as an example where a subtly-wrong idea could cause cascading problems.

simonw 2y ago

I don't worry about that much at all, because my experience of learning is that you inevitably have to reconsider the fundamentals pretty often as you go along. High school science is a great example: once you get to university you have to un-learn all sorts of things that you learned earlier because they were simplifications that no longer apply. Terry Pratchett has a great quote about this: https://simonwillison.net/2024/Jul/1/terry-pratchett/ https://simonwillison.net/2024/Jul/1/terry-pratchett/ For fields that I'm completely new to, the thing I need most is a grounding in the rough shape and jargon of the field. LLMs are fantastic at that - it's then up to me to take that grounding and those jargon terms and start building my own accurate-as-possible mental model of how that field actually works. If you treat LLMs as just one unreliable source of information (like your well-read friend who's great at explaining things in terms that you understand but may not actually be a world expert on a subject) you can avoid many of the pitfalls. Where things go wrong is if you assume LLMs are a source of irrefutable knowledge.

lolinder 2y ago
> like your well-read friend who's great at explaining things in terms that you understand but may not actually be a world expert on a subject I guess part of my problem with using them this way is that I am that well-read friend. I know how the sausage is made, how easy it is to bluff a response to any given question, and for myself I tend to prefer reading original sources to ensure that the understanding that I'm conveying is as accurate as I can make it and not a third-hand account whose ultimate source is a dubious Reddit thread. > High school science is a great example: once you get to university you have to un-learn all sorts of things that you learned earlier because they were simplifications that no longer apply. The difference between this and a bad mental model generated by an LLM is that the high school science models were designed to be good didactic tools and to be useful abstractions in their own right. An LLM output may be neither of those.
- simonw 2y ago
  If you "tend to prefer reading original sources" then I think you're the best possible candidate for LLM-assisted learning, because you'll naturally use them as a starting point, not the destination. I like to use LLMs to get myself the grounding I need to then start reading further around a topic from more reliable sources. That's a great point about high school models being deliberately designed as didactic tools. LLMs will tend to spit those out too, purely because the high school version of anything has been represented heavily enough in the training data that it's more likely than not to fall out of the huge matrix of numbers!
  - lolinder 2y ago
    > LLMs will tend to spit those out too, purely because the high school version of anything has been represented heavily enough in the training data that it's more likely than not to fall out of the huge matrix of numbers! That assumes that the high school version of the subject exists, which is unlikely because I already have the high school version of most subjects that have a high school version. The subjects that I would want to dig into at that level would be something along the lines of chemical engineering, civil engineering, or economics—subjects that I don't yet know very much about but have interest or utility for me. These subjects don't have a widely-taught high school version crafted by humans, and I don't trust that they would have enough training data to produce useful results from an LLM.
    - fragmede 2y ago
      at what point does a well-read high school-level LLM graduate to college? I asked one about Reinforcement Learning, and at first it treated me like the high schooler, but I was able to prod it into giving me answers more suitable for my level. Of course, I don't know what's hallucinated or not, but it satisfied my curiosity enough to be worth my while. I'm not looking to change careers, so getting things 100% right about in the fields of chemical engineering, civil engineering, or economics isn't necessary. I look at it as the same way I think of astrophysics. After reading Steven Hawkins book, I still don't really know astrophysics at all, but I have a good enough model of things. And as they say, all models are wrong, some are useful. If I were a lawyer using these things for work, I'd be insane to trust one at this stage, but the reality is I'm not using my digging into things I don't know about for anything load bearing, but even if I were, I'd still use an LLM to get started. Eg the post didn't state how the author learned anything the name for the dropped letter O, but I can describe a thing and have the LLM give me the name of it. there's an emphasis on getting things totally 100% right does erode trust, but you get a sense for what's could be a hallucination and then check background resources if you get enough experience with the tool.

safetytrick 2y ago

In the article the author mentions wanting to benchmark a GPU and using ChatGPT to write CUDA. Benchmarks are easy to mess up and to interpret incorrectly without understanding. I see this as an example where a subtly-wrong idea could cause cascading problems.