3 ms·

Recently, Gemini helped me fix a bug in a PPP driver (Zephyr OS) without prior knowledge of PPP or even driver development really. I would copy-paste logs of ra

Recently, Gemini helped me fix a bug in a PPP driver (Zephyr OS) without prior knowledge of PPP or even driver development really. I would copy-paste logs of raw PPP frames in HEX and it would just decode everything and explain the meaning of each bytes. In about an hour, I knew enough about PPP to fix the bug and submit a patch. https://g.co/gemini/share/7edf8fa373fe https://g.co/gemini/share/7edf8fa373fe

Helmut10001 1y ago
Yes, it fells like setting the `-h` flag for logs (human readable).
skydhash 1y ago
Or you could just read the PPP RFC [0]. I’m not saying that your approach is wrong. But most LLM workflows are either brute forcing the solution, or seeking a local minima to be stuck in. It’s like doing thousands of experiments of objects falling to figure out gravity while there’s a physics textbooks nearby. [0]: https://datatracker.ietf.org/doc/html/rfc1661 https://datatracker.ietf.org/doc/html/rfc1661
- olalonde 1y ago
  Ironically, I could’ve read all 50 pages of that RFC and still missed the actual issue. What really helped was RFC 1331[0], specifically the "Async-Control-Character-Map" section. That said, I’m building a product - not a PPP driver - so the quicker I can fix the problem and move on, the better. [0] https://datatracker.ietf.org/doc/html/rfc1331 https://datatracker.ietf.org/doc/html/rfc1331
- wrasee 1y ago
  I could also walk everywhere, but sometimes technology can help. There’s no way I could fully read that RFC in an hour. And that’s before you even know what reading to focus your attention on, so you’re just being a worse LLM at that point.
  - Retric 1y ago
    The difference is you’d remember some of the context from reading the thing where an LLM is starting from scratch every single time it comes up.
    - cgriswald 1y ago
      There are opportunity costs to consider along with relevance. Suppose you are staying at my place. Are you going to read the manual for my espresso machine in total or are you going to ask me to show you how to use it or make one for you? In any case, LLMs are not magical forgetfulness machines. You can use a calculator to avoid learning arithmetic but using a calculator doesn’t necessitate failing to learn arithmetic. You can ask a question of a professor or fellow student, but failing to read the textbook to answer that question doesn’t necessitate failing to develop a mental model or incorporate the answer into an existing one. You can ask an LLM a question and blindly use its answer but using an LLM doesn’t necessitate failing to learn.
      - Retric 1y ago
        There’s plenty to learn from using LLM’s including how to interact with an LLM. However, even outside of using a LLM the temptation is always to keep the blinders on do a deep dive for a very specific bug and repeat as needed. It’s the local minima of effort and very slowly you do improve as those deep dives occasionally come up again, but what keeps it from being a global minimum is these systems aren’t suddenly going away. It’s not a friend’s expresso machine, it’s now sitting in your metaphorical kitchen. As soon as you’re dealt with say a CSS bug the odds of seeing another in the future are dramatically higher. Thus optimizing for diminishing returns means spending a few hours learning the basics of any system or protocol you encounter is just a useful strategy. If you spend 1% of your time on a strategy that makes you 2% more efficient that’s a net win.
        skydhash 1y ago
        Sometimes learning means understanding, aka a deep dive on the domain. Only a few domains are worth that. For the others, it's only about placing landmark so you can quickly recognize a problem and find the relevant information before solving it. I believe the best use case of LLMs is when you have recognized the problem and know the general shape of the solution, but have no time to wrangle the specifics of the implementation. So you can provide the context and its constraint in order to guide the LLM's generation, as well as recognize wrong outputs. But that's not learning or even problem's solving. It's just a time saving trick. And one that's not reliable. And the fact is that there's a lot of information about pretty much anything. But I see people trying to skip the foundation (not glamorous enough, maybe) and go straight for the complicated stuff. And LLMs are good for providing the illusion that it can be the right workflow.
tralarpa 1y ago
Interesting that it works for you. I tried several times something similar with frames from a 5G network and it mixed fields from 4G and 5G in its answers (or even from non-cellular network protocols because they had similar features as the 5G protocol I was looking at). Occasionally, the explanation was completely invented or based on discussions of planned features for future versions. I have really learned to mistrust and double check every single line those systems produce. Same for writing code. Everything they produce looks nice and reasonable on the surface but when you dig deaper it falls apart unless it's something very very basic.
- foobarian 1y ago
  Similarly I found the results pretty mixed whenever a library or framework with a lot of releases/versions is involved. The LLM tends to mix and match features from across versions.

Helmut10001 1y ago

Yes, it fells like setting the `-h` flag for logs (human readable).

skydhash 1y ago

Or you could just read the PPP RFC [0]. I’m not saying that your approach is wrong. But most LLM workflows are either brute forcing the solution, or seeking a local minima to be stuck in. It’s like doing thousands of experiments of objects falling to figure out gravity while there’s a physics textbooks nearby. [0]: https://datatracker.ietf.org/doc/html/rfc1661 https://datatracker.ietf.org/doc/html/rfc1661

olalonde 1y ago
Ironically, I could’ve read all 50 pages of that RFC and still missed the actual issue. What really helped was RFC 1331[0], specifically the "Async-Control-Character-Map" section. That said, I’m building a product - not a PPP driver - so the quicker I can fix the problem and move on, the better. [0] https://datatracker.ietf.org/doc/html/rfc1331 https://datatracker.ietf.org/doc/html/rfc1331
wrasee 1y ago
I could also walk everywhere, but sometimes technology can help. There’s no way I could fully read that RFC in an hour. And that’s before you even know what reading to focus your attention on, so you’re just being a worse LLM at that point.
- Retric 1y ago
  The difference is you’d remember some of the context from reading the thing where an LLM is starting from scratch every single time it comes up.
  - cgriswald 1y ago
    There are opportunity costs to consider along with relevance. Suppose you are staying at my place. Are you going to read the manual for my espresso machine in total or are you going to ask me to show you how to use it or make one for you? In any case, LLMs are not magical forgetfulness machines. You can use a calculator to avoid learning arithmetic but using a calculator doesn’t necessitate failing to learn arithmetic. You can ask a question of a professor or fellow student, but failing to read the textbook to answer that question doesn’t necessitate failing to develop a mental model or incorporate the answer into an existing one. You can ask an LLM a question and blindly use its answer but using an LLM doesn’t necessitate failing to learn.
    - Retric 1y ago
      There’s plenty to learn from using LLM’s including how to interact with an LLM. However, even outside of using a LLM the temptation is always to keep the blinders on do a deep dive for a very specific bug and repeat as needed. It’s the local minima of effort and very slowly you do improve as those deep dives occasionally come up again, but what keeps it from being a global minimum is these systems aren’t suddenly going away. It’s not a friend’s expresso machine, it’s now sitting in your metaphorical kitchen. As soon as you’re dealt with say a CSS bug the odds of seeing another in the future are dramatically higher. Thus optimizing for diminishing returns means spending a few hours learning the basics of any system or protocol you encounter is just a useful strategy. If you spend 1% of your time on a strategy that makes you 2% more efficient that’s a net win.
      - skydhash 1y ago
        Sometimes learning means understanding, aka a deep dive on the domain. Only a few domains are worth that. For the others, it's only about placing landmark so you can quickly recognize a problem and find the relevant information before solving it. I believe the best use case of LLMs is when you have recognized the problem and know the general shape of the solution, but have no time to wrangle the specifics of the implementation. So you can provide the context and its constraint in order to guide the LLM's generation, as well as recognize wrong outputs. But that's not learning or even problem's solving. It's just a time saving trick. And one that's not reliable. And the fact is that there's a lot of information about pretty much anything. But I see people trying to skip the foundation (not glamorous enough, maybe) and go straight for the complicated stuff. And LLMs are good for providing the illusion that it can be the right workflow.

tralarpa 1y ago

Interesting that it works for you. I tried several times something similar with frames from a 5G network and it mixed fields from 4G and 5G in its answers (or even from non-cellular network protocols because they had similar features as the 5G protocol I was looking at). Occasionally, the explanation was completely invented or based on discussions of planned features for future versions. I have really learned to mistrust and double check every single line those systems produce. Same for writing code. Everything they produce looks nice and reasonable on the surface but when you dig deaper it falls apart unless it's something very very basic.

foobarian 1y ago
Similarly I found the results pretty mixed whenever a library or framework with a lot of releases/versions is involved. The LLM tends to mix and match features from across versions.