5 ms·

Heavily down voted, which is fair because I didn't really explain what I meant, which was: Would using LLM's to parse the generated diffs, as a first pass, be u

Heavily down voted, which is fair because I didn't really explain what I meant, which was: Would using LLM's to parse the generated diffs, as a first pass, be useful/efficient for spotting and interpreting discrepancies?

arccy 2y ago
when your goal is to improve security, the unreliability that comes with LLMs is not the answer.
- chipdart 2y ago
  I don't think this is a relevant take. Your goal is to implement a system to automatically scan countless packages and run a heuristic to determine if a package is suspicious or not. You're complaining about false positives/false negatives while ignoring that packages that not checking packages at all is not an improvement.
  - Zambyte 2y ago
    Personally I think using LLMs to scan is a good idea, but an honesty negative is a potential false sense of security. I think using LLMs here are useful for finding unintentional security flaws. I don't think it's a great tool to find intentional security flaws a la the xz situation. People might be less inclined to dig into the code directly if it was stamped with a green check mark by a GPT.
- Aeolun 2y ago
  Not necessarily, it reduces false positives. It just doesn’t do anything for false negatives (arguably makes the problem worse). If you just want to see if there is incidence of valid differences, this seems fine. But I wouldn’t use it as a guarantee.
- seoulmetro 2y ago
  Neither does grabbing yet another online third party's untested data?
- SubiculumCode 2y ago
  Using machine learning, including LLMs, to detect and mitigate malicious code is of interest to a whole lot of smarter people than me, really suggests your flippant rejection of their potential is premature. https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.19715 https://arxiv.org/abs/2404.19715 https://www.sciencedirect.com/science/article/pii/S2666389923001241 https://www.sciencedirect.com/science/article/pii/S266638992...
pornel 2y ago
It could work for classifying honest/innocent differences. However, LLMs are incredibly naive, so they could be easily fooled by a malicious actor (probably as easy as adding a comment that this is definitely NOT a backdoor).
- SubiculumCode 2y ago
  LLMs are broadly naive, but when fine tuned on a small domain of expertise/knowledge, this problem is less impactful.

arccy 2y ago

when your goal is to improve security, the unreliability that comes with LLMs is not the answer.

chipdart 2y ago
I don't think this is a relevant take. Your goal is to implement a system to automatically scan countless packages and run a heuristic to determine if a package is suspicious or not. You're complaining about false positives/false negatives while ignoring that packages that not checking packages at all is not an improvement.
- Zambyte 2y ago
  Personally I think using LLMs to scan is a good idea, but an honesty negative is a potential false sense of security. I think using LLMs here are useful for finding unintentional security flaws. I don't think it's a great tool to find intentional security flaws a la the xz situation. People might be less inclined to dig into the code directly if it was stamped with a green check mark by a GPT.
Aeolun 2y ago
Not necessarily, it reduces false positives. It just doesn’t do anything for false negatives (arguably makes the problem worse). If you just want to see if there is incidence of valid differences, this seems fine. But I wouldn’t use it as a guarantee.
seoulmetro 2y ago
Neither does grabbing yet another online third party's untested data?
SubiculumCode 2y ago
Using machine learning, including LLMs, to detect and mitigate malicious code is of interest to a whole lot of smarter people than me, really suggests your flippant rejection of their potential is premature. https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2405.17238 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.02056 https://arxiv.org/abs/2404.19715 https://arxiv.org/abs/2404.19715 https://www.sciencedirect.com/science/article/pii/S2666389923001241 https://www.sciencedirect.com/science/article/pii/S266638992...

pornel 2y ago

It could work for classifying honest/innocent differences. However, LLMs are incredibly naive, so they could be easily fooled by a malicious actor (probably as easy as adding a comment that this is definitely NOT a backdoor).

SubiculumCode 2y ago
LLMs are broadly naive, but when fine tuned on a small domain of expertise/knowledge, this problem is less impactful.