3 ms·

> But, truly, what is the solution? Let's use GitHub as an example. We have forks, and stars. Maybe we could also have some kind of build endorsement? How one

> But, truly, what is the solution? Let's use GitHub as an example. We have forks, and stars. Maybe we could also have some kind of build endorsement? How one would verify that the endorser is worth your trust, I am not entirely sure. Maybe endorsers could eventually be rated by CVEs found in their endorsements, and that would build trust?

achempion 2y ago
Copilot for sure should be able to describe the code and spot basic malware
- consumer451 2y ago
  LLM or human, what if they both competed in some sort of "I have the least CVEs in my endorsements list" battle? This would actually be an excellent LLM coding benchmark,[0] in addition to a human endorser benchmark. [0] If nobody is already doing this, especially retrospectively, and you do, then please at least give me a shout out. :)
- junon 2y ago
  No it won't. I could write you very basic, obvious malware that is obfuscated just enough for copilot to miss it 100% of the time. Let alone things like what JiaTan wrote.
moi2388 2y ago
How about directly linking to the CVEs and how quickly they were mitigated and in which commit? Pay researchers to analyse repos without any. Post results. Link to the repo with mitigation PRs. It’s insane this isn’t the standard already
- consumer451 2y ago
  > Pay researchers to analyse repos without any. This is the problem, the best we can do is pay via exposure. But that actually ain't nothing. Not just individuals, but also orgs could then make money from private contracts based on their reputations? This should be the benchmark of trust. Could there be anything better?
swinglock 2y ago
Are you certain that CVEs are a good indicator?
- consumer451 2y ago
  Excellent question. No, I am not. I am just attempting to use my very limited knowledge on this subject to hopefully further discussion on a topic that feels really important. I would love other people to jump in and elevate this conversation. Sure, CVEs might not be the ideal metric. Could you, or anyone else, suggest a better metric? If GitHub is too lethargic to do even contemplate this type of change, maybe this could be a differentiator for GitLab?
alp1n3_eth 2y ago
They could build an optional "risk score" that open-source community-oriented projects could turn on. It could include requirements like having something dependabot-esque along with CodeQL enabled. Rules could be created for CodeQL (if they haven't already) that check for obfuscated code, suspicious access (keychain, password storage, etc.) and other items. On top of that it could have forced release binary scanning via VirusTotal/insert-malware-scanning-vendor-here.

achempion 2y ago

Copilot for sure should be able to describe the code and spot basic malware

consumer451 2y ago
LLM or human, what if they both competed in some sort of "I have the least CVEs in my endorsements list" battle? This would actually be an excellent LLM coding benchmark,[0] in addition to a human endorser benchmark. [0] If nobody is already doing this, especially retrospectively, and you do, then please at least give me a shout out. :)
junon 2y ago
No it won't. I could write you very basic, obvious malware that is obfuscated just enough for copilot to miss it 100% of the time. Let alone things like what JiaTan wrote.

moi2388 2y ago

How about directly linking to the CVEs and how quickly they were mitigated and in which commit? Pay researchers to analyse repos without any. Post results. Link to the repo with mitigation PRs. It’s insane this isn’t the standard already

consumer451 2y ago
> Pay researchers to analyse repos without any. This is the problem, the best we can do is pay via exposure. But that actually ain't nothing. Not just individuals, but also orgs could then make money from private contracts based on their reputations? This should be the benchmark of trust. Could there be anything better?

swinglock 2y ago

Are you certain that CVEs are a good indicator?

consumer451 2y ago
Excellent question. No, I am not. I am just attempting to use my very limited knowledge on this subject to hopefully further discussion on a topic that feels really important. I would love other people to jump in and elevate this conversation. Sure, CVEs might not be the ideal metric. Could you, or anyone else, suggest a better metric? If GitHub is too lethargic to do even contemplate this type of change, maybe this could be a differentiator for GitLab?

alp1n3_eth 2y ago

They could build an optional "risk score" that open-source community-oriented projects could turn on. It could include requirements like having something dependabot-esque along with CodeQL enabled. Rules could be created for CodeQL (if they haven't already) that check for obfuscated code, suspicious access (keychain, password storage, etc.) and other items. On top of that it could have forced release binary scanning via VirusTotal/insert-malware-scanning-vendor-here.