4 ms·

I believe this is something that automated code parsing (a la Github Copilot) could really shine. Those tools are able to explain what a block of code does, so

I believe this is something that automated code parsing (a la Github Copilot) could really shine. Those tools are able to explain what a block of code does, so it should be possible to catch many types of malware hidden in source code.

qbasic_forever 4y ago
If a human can't figure it out than an AI trained on human behavior (or more specifically, trained on comments left next to code) can't figure it out either. It's not like malware authors annotate their code with nice "here's where I exfiltrate all the user's secret data" comments.
- jart 4y ago
  That's not going to stop people from trying though. If people keep abusing the package managers like this, then publishing open source code is at risk of ending up like distributing binaries on Windows, where you need to fight dozens of virus scanners arbitrarily blocking your code for no apparent reason. Imagine opening a tab in VSCode only to see a popup warning saying it might be harmful, even though you just wrote it yourself.
  - imwillofficial 4y ago
    This kinda happens already. I get prompted to trust stuff I wrote
  - dapids 4y ago
    That's exactly was vscode does for me.
    - jart 4y ago
      Are you talking about this? https://code.visualstudio.com/docs/editor/workspace-trust https://code.visualstudio.com/docs/editor/workspace-trust That's different from how virus scanners work and not exactly what I'm talking about. Even Emacs does that.
    - ElectricalUnion 4y ago
      That is different, vscode finally realized most people don't want stuxnet-class implicit autorun behaviour from random files and folders in your system.
- selcuka 4y ago
  I never claimed that it would be able to catch all malware, but it beats a human team reading through hundreds or thousands lines of code, for every package, after every update. I believe it can easily point out many types of suspicious additions to existing code.
- rurp 4y ago
  A focused human will currently do better than automated tools, but actually getting a human to focus on this sort of thing is rare. Most projects I'm familiar with use a multitude of dependencies and only do cursory verification of changes at best. An automated tool to check all depency changes for suspicious code and flag that to a human could be valuable. Whether or not that could be done in a way where wading through the false positives is worth it, I'm not sure, but it's a reasonable idea.
323 4y ago
This is the kind of stuff that antiviruses did for ages, from signature, code simulation to heuristics. They have a lot of expertise in this, it feels like they could branch out in finding malware in source code, instead of in binaries.
- Beltiras 4y ago
  Binaries can be seen as "source code" for the CPU.
- devonkim 4y ago
  Source code has issues with obfuscation methodologies that can defeat a lot of techniques. It’s why companies are trying to build more analysis down into the kernel such as via EBPF. For example, concatenating a series of strings and characters that wind up reading from .AWS/credentials in the end is surprisingly tough to catch based upon simple pattern recognition alone, especially if it’s done in a subtle way such as with a spare buffer while doing other legit activities. So until the syscall gets issued and all substitutions resolved the user space analysis can be highly resource intensive or inaccurate
  - jhugo 4y ago
    Right, code analysis to try to detect places where it reads from ~/.aws/credentials is never going to be reliable. The correct approach is not to run untrusted code in an environment where it can read your AWS credentials.
    - tremon 4y ago
      And if that's infeasible, to not run untrusted code in an environment where it has unfettered outbound access to the Internet.
    - devonkim 4y ago
      Another approach is to only run credentials that are ephemeral, which is sort of what most SSO systems will do for cloud IAM. Instance profiles using IDMSv2 work as well, too. However some malware out there only needs a few seconds of dwell time to wreak some serious havoc so even ephemeral credentials may as well be the same as static credentials potentially, especially if your credentials are used to do permanent privilege escalation. All it really can do then is provide a time window of usage and make filtering through a SIEM much more accurate, which is certainly valuable for forensics at the very least and even more important in terms of law (chain of custody, irrefutability, etc).
rjh29 4y ago
Isn't this impossible as per the halting problem? It would be better to run the code in a sandbox and log all the syscalls it makes, similar to how Cuckoo's malware sandbox works.
- Scea91 4y ago
  Halting problem is only relevant if we are discussing a perfect 100% solution with no false-positives and false-negatives. In real-life it is not of much relevance. I haven't seen a practical case in my life where I would conclude that it is undecidable to say what a function does (happy to see some practical examples if someone has some). For a theoretical program that "uploads credentials iff a sub-program halts" you would probably block it and live with a possible false-positive.
  - PeterisP 4y ago
    > undecidable to say what a function does (happy to see some practical examples if someone has some) Any and all functions that involve templating/plugins/metaprogramming/etc (common in various web frameworks) that can effectively run arbitrary code and thus are undecidable. Also any function that does deserialization and thus will instantiate novel objects with their code, e.g. Python code that uses pickle will often be undecidable as its execution will depend on what exactly is unpickled.

qbasic_forever 4y ago

If a human can't figure it out than an AI trained on human behavior (or more specifically, trained on comments left next to code) can't figure it out either. It's not like malware authors annotate their code with nice "here's where I exfiltrate all the user's secret data" comments.

jart 4y ago
That's not going to stop people from trying though. If people keep abusing the package managers like this, then publishing open source code is at risk of ending up like distributing binaries on Windows, where you need to fight dozens of virus scanners arbitrarily blocking your code for no apparent reason. Imagine opening a tab in VSCode only to see a popup warning saying it might be harmful, even though you just wrote it yourself.
- imwillofficial 4y ago
  This kinda happens already. I get prompted to trust stuff I wrote
- dapids 4y ago
  That's exactly was vscode does for me.
  - jart 4y ago
    Are you talking about this? https://code.visualstudio.com/docs/editor/workspace-trust https://code.visualstudio.com/docs/editor/workspace-trust That's different from how virus scanners work and not exactly what I'm talking about. Even Emacs does that.
  - ElectricalUnion 4y ago
    That is different, vscode finally realized most people don't want stuxnet-class implicit autorun behaviour from random files and folders in your system.
selcuka 4y ago
I never claimed that it would be able to catch all malware, but it beats a human team reading through hundreds or thousands lines of code, for every package, after every update. I believe it can easily point out many types of suspicious additions to existing code.
rurp 4y ago
A focused human will currently do better than automated tools, but actually getting a human to focus on this sort of thing is rare. Most projects I'm familiar with use a multitude of dependencies and only do cursory verification of changes at best. An automated tool to check all depency changes for suspicious code and flag that to a human could be valuable. Whether or not that could be done in a way where wading through the false positives is worth it, I'm not sure, but it's a reasonable idea.

323 4y ago

This is the kind of stuff that antiviruses did for ages, from signature, code simulation to heuristics. They have a lot of expertise in this, it feels like they could branch out in finding malware in source code, instead of in binaries.

Beltiras 4y ago
Binaries can be seen as "source code" for the CPU.
devonkim 4y ago
Source code has issues with obfuscation methodologies that can defeat a lot of techniques. It’s why companies are trying to build more analysis down into the kernel such as via EBPF. For example, concatenating a series of strings and characters that wind up reading from .AWS/credentials in the end is surprisingly tough to catch based upon simple pattern recognition alone, especially if it’s done in a subtle way such as with a spare buffer while doing other legit activities. So until the syscall gets issued and all substitutions resolved the user space analysis can be highly resource intensive or inaccurate
- jhugo 4y ago
  Right, code analysis to try to detect places where it reads from ~/.aws/credentials is never going to be reliable. The correct approach is not to run untrusted code in an environment where it can read your AWS credentials.
  - tremon 4y ago
    And if that's infeasible, to not run untrusted code in an environment where it has unfettered outbound access to the Internet.
  - devonkim 4y ago
    Another approach is to only run credentials that are ephemeral, which is sort of what most SSO systems will do for cloud IAM. Instance profiles using IDMSv2 work as well, too. However some malware out there only needs a few seconds of dwell time to wreak some serious havoc so even ephemeral credentials may as well be the same as static credentials potentially, especially if your credentials are used to do permanent privilege escalation. All it really can do then is provide a time window of usage and make filtering through a SIEM much more accurate, which is certainly valuable for forensics at the very least and even more important in terms of law (chain of custody, irrefutability, etc).

rjh29 4y ago

Isn't this impossible as per the halting problem? It would be better to run the code in a sandbox and log all the syscalls it makes, similar to how Cuckoo's malware sandbox works.

Scea91 4y ago
Halting problem is only relevant if we are discussing a perfect 100% solution with no false-positives and false-negatives. In real-life it is not of much relevance. I haven't seen a practical case in my life where I would conclude that it is undecidable to say what a function does (happy to see some practical examples if someone has some). For a theoretical program that "uploads credentials iff a sub-program halts" you would probably block it and live with a possible false-positive.
- PeterisP 4y ago
  > undecidable to say what a function does (happy to see some practical examples if someone has some) Any and all functions that involve templating/plugins/metaprogramming/etc (common in various web frameworks) that can effectively run arbitrary code and thus are undecidable. Also any function that does deserialization and thus will instantiate novel objects with their code, e.g. Python code that uses pickle will often be undecidable as its execution will depend on what exactly is unpickled.