3 ms·

This represents a massive failure of the Open Source model and it's worth thinking carefully about as more and more people advocate for fully open source AI mod

This represents a massive failure of the Open Source model and it's worth thinking carefully about as more and more people advocate for fully open source AI models. People could spend a lot of time on token safety features only to have them backdoored by a sneaky PR and it's not clear what level of damage this could cause in the near future.

mikedelfino 3y ago
Isn't it the opposite, though? The whole thing was caught precisely because it was open source. Sure, you can argue that the vulnerability was introduced by a contributor, but what prevents the same thing from happening in a private repository, perpetrated by an actual employee? Especially if you consider that many vulnerabilities can be introduced with code more likely to pass as legitimate mistakes.
- bombcar 3y ago
  Or think about the closed-source products that use open source code (basically everything these days) - had this targeted versions of xz building on Windows only it might have lain dormant longer (not sure this attacks would have been useful, but something analogous could be).
- humansareok1 3y ago
  But it wasn't caught because it was open source? It was caught basically by chance by an engineer using the distro who admits they only caught it because of a series of coincidences.
smikesix 3y ago
thats not an issue with open source, but an issue with trust and structure. in closed source it can also happen and it would be even harder to spot, since you have no clue what is going on and there would be maybe 1-2 ppl in the whole world that could identify that issue. with open source anyone can. the joke is that he found it because the ssh login felt too long. this dude needs a medal. what needs to be controlled are system relevant libraries that open up such possibilites and also not including not code data into builds. any binary blob is bad, anything that cant simply be read is bad. if you put in code, ppl will control it and most of the time see what it does. but if you hide the real code in a binary blob, such files should not be included in builds, but shipped separately

mikedelfino 3y ago

Isn't it the opposite, though? The whole thing was caught precisely because it was open source. Sure, you can argue that the vulnerability was introduced by a contributor, but what prevents the same thing from happening in a private repository, perpetrated by an actual employee? Especially if you consider that many vulnerabilities can be introduced with code more likely to pass as legitimate mistakes.

bombcar 3y ago
Or think about the closed-source products that use open source code (basically everything these days) - had this targeted versions of xz building on Windows only it might have lain dormant longer (not sure this attacks would have been useful, but something analogous could be).
humansareok1 3y ago
But it wasn't caught because it was open source? It was caught basically by chance by an engineer using the distro who admits they only caught it because of a series of coincidences.

smikesix 3y ago

thats not an issue with open source, but an issue with trust and structure. in closed source it can also happen and it would be even harder to spot, since you have no clue what is going on and there would be maybe 1-2 ppl in the whole world that could identify that issue. with open source anyone can. the joke is that he found it because the ssh login felt too long. this dude needs a medal. what needs to be controlled are system relevant libraries that open up such possibilites and also not including not code data into builds. any binary blob is bad, anything that cant simply be read is bad. if you put in code, ppl will control it and most of the time see what it does. but if you hide the real code in a binary blob, such files should not be included in builds, but shipped separately