3 ms·
How does it "reason" though, I thought LLM just generated likely next words?
by WhiteBlueSkies 4y ago
How does it "reason" though, I thought LLM just generated likely next words?
- dqpb 4y agoMaybe that is a form of reasoning.
- ramoz 4y agoI'm no expert and don't have a proper answer. But my hunches/experience is that `proper prompting + nature of code being logical` really showcases the power of whatever statistical distribution and alignment occurs during generation. Further, there must be major upstream efforts of high-quality training data curation + creation, an advanced training tricks like using a LLM's strong ability in one task area to support the training of an area it is weak in.
- ramoz 4y agoJust saw this: https://twitter.com/BoshiWang2/status/1605405235677040641?s=20&t=RHLyM0BMiCrayOvlAl2vOg https://twitter.com/BoshiWang2/status/1605405235677040641?s=...
- meken 4y agoMy understanding is the transformer layer in the LLM is basically doing something akin to message passing, it’s like a mini computer. In predicting the next word, it has to understand a lot about a lot of different kinds of topics My understanding is kinda fuzzy because I haven’t coded it up myself, but this was the takeaway I got from this explanation (starts at 36:21) https://youtu.be/cdiD-9MMpb0 https://youtu.be/cdiD-9MMpb0