4 ms·

Google can look up into their index and can remove whatever they want to, within minutes. But how that can be possible for an LLM? That is, "decontaminate" the

by wg0 3y ago

Google can look up into their index and can remove whatever they want to, within minutes. But how that can be possible for an LLM? That is, "decontaminate" the model from certain parts of the corups? I can only think of excluding the data set from the training and then retrain? As a side note, I think LLM frenzy would be dead in few years, 10 years time frame at max. The rent seeking on these LLMs as of today would no more be a viable or as profitable business model as more inference circuitry gets out in the wild into laptops and phones, more models get released, tweaked by the community and such. People thinking to downvote and dismiss this should see the history of commercial Unix and how that turned out to be today and how almost no workload (other than CAD, Graphics) runs on Windows or Unix including this very forum, I highly doubt is hosted on Windows or a commercial variant of Unix.

pxoe 3y ago
maybe they should build a better LLM? maybe they could ask the AI to make a better system. after all, tech and ai is so powerful that they could do virtually anything, except having accountability as it turns out.
Firaxus 3y ago
But, wasn’t the reason proprietary unixes died out at major work horses because of a nearly feature comparable free alternative (Linux)? Extending the analogy, LLMs won’t die out, just proprietary ones. (Which is where I think this tech will actually go anyway.)
- wg0 3y ago
  LLMs won't die out but proprietary LLMs behind APIs might not have valuations of hundreds of billions of dollars. Crowd source, crowed trained (distributed training) fast enough, good enough generative models that are updated (and downloadable) every few months would start to erode the subscriber base gradually. I might be very very wrong here but it seems like so from where I see it.
- deleted 3y ago
  [deleted]
iandanforth 3y ago
You may be interested in https://unlearning-challenge.github.io/ https://unlearning-challenge.github.io/
delta_p_delta_x 3y ago
> almost no workload (other than CAD, Graphics) runs on Windows or Unix including this very forum About a fifth to a quarter of public-facing Web servers are Windows Server. Most famously, Stack Overflow[1]. [1]: https://meta.stackexchange.com/a/10370/1424704 https://meta.stackexchange.com/a/10370/1424704
- candiddevmike 3y ago
  > About a fifth to a quarter of public-facing Web servers are Windows Server Got a link for that? Best I can find is 5% of all websites: https://www.netcraft.com/blog/may-2023-web-server-survey/ https://www.netcraft.com/blog/may-2023-web-server-survey/
- wg0 3y ago
  20% of workloads running on Windows should result in corresponding number of jobs as well but that's not what I see. Most companies are writing software with software developed on Linux first and for Linux first (or Unix) and later ported to Windows as an after thought. I'm thinking Python, Ruby, NodeJS, Rust, Go, Java, PHP but not seeing as much of C#/ASP.NET which should at least be 20% of the market? Only two explanations - either I am in a social bubble so don't have exposure or writing software for Windows is so much easy that it takes five times less engineering muscle.
  - argiopetech 3y ago
    There are plenty of .NET jobs, and .NET (Core, particularly) is really easy to write. That said, I'd guess the difference is that the startup and big tech world (i.e., "software companies") like our fancy stacks, but non-software companies prefer stability and familiarity. It makes way more sense for most companies to have a 3-man "bespoke software" department (sys/db admin, sr engineer, jr engineer) on a stack supported by a big company (Microsoft) where most of the work is maintenance and the position lasts an entire career. It's a big enough team to support most small to middling businesses, but not so big that the push to rewrite everything in [language/framework of the week] gains traction. The practical conclusion is that these companies have few spots to fill, and they probably don't advertise where you're looking.
deleted 3y ago
[deleted]
wegfawefgawefg 3y ago
Unix kinda still does the same thing now as before. Future big ai models might be totally different in quality, and latency.
_jab 3y ago
The argument may be that having very large models that everyone uses is a bad idea, and that companies and even individuals should instead be empowered to create their own smaller models, trained on data they trust. This will only become more feasible as the technology progresses.
danaris 3y ago
> But how that can be possible for an LLM? Well, it seems to me that's part of the problem here. And it's their problem, one they created for themselves by just assuming they could safely take absolutely every bit of data they could get their hands on to train their models.
blovescoffee 3y ago
People thinking to dismiss this should, period. Consider that Open AI and similar companies are the only ones in the AI space with the market cap to build out profitable hardware projects which open source can't. Or maybe every investor is just dumb and likes throwing millions of dollars away so they can participate in a hype train.
bigbillheck 3y ago
> But how that can be possible for an LLM? They should have thought of that before they went ahead and trained on whatever they could get. Image models are going to have similar problems, even if they win on copyright there's still CSAM in there: https://www.theregister.com/2023/12/20/csam_laion_dataset/ https://www.theregister.com/2023/12/20/csam_laion_dataset/
015a 3y ago
Windows and MacOS (and their closed source derivatives) are probably at least as large as Linux, even including all the servers Linux is deployed on. Proprietary UNIX did not "die out"; Apple sells about a quarter million of them every year. The majority of the world's computing systems runs on closed source software. Believing the opposite is bubble-thinking. Its not just Windows/MacOS. Most Android distros are not actually open source. Power control systems. Traffic control systems. Networking hardware. Even the underlying operating systems which power the VMs you run on AWS that are technically open source. The billions of little computers that form together to make the modern world work; they're mostly closed source.
benrow 3y ago
Google have their "Machine Unlearning" challenge to address this specific issue - removing the influence of given training data without retraining from scratch. Seems like a hard problem. https://blog.research.google/2023/06/announcing-first-machine-unlearning.html?m=1 https://blog.research.google/2023/06/announcing-first-machin...

by wg0 3y ago

pxoe 3y ago

maybe they should build a better LLM? maybe they could ask the AI to make a better system. after all, tech and ai is so powerful that they could do virtually anything, except having accountability as it turns out.

Firaxus 3y ago

But, wasn’t the reason proprietary unixes died out at major work horses because of a nearly feature comparable free alternative (Linux)? Extending the analogy, LLMs won’t die out, just proprietary ones. (Which is where I think this tech will actually go anyway.)

wg0 3y ago
LLMs won't die out but proprietary LLMs behind APIs might not have valuations of hundreds of billions of dollars. Crowd source, crowed trained (distributed training) fast enough, good enough generative models that are updated (and downloadable) every few months would start to erode the subscriber base gradually. I might be very very wrong here but it seems like so from where I see it.
deleted 3y ago
[deleted]

iandanforth 3y ago

You may be interested in https://unlearning-challenge.github.io/ https://unlearning-challenge.github.io/

delta_p_delta_x 3y ago

> almost no workload (other than CAD, Graphics) runs on Windows or Unix including this very forum About a fifth to a quarter of public-facing Web servers are Windows Server. Most famously, Stack Overflow[1]. [1]: https://meta.stackexchange.com/a/10370/1424704 https://meta.stackexchange.com/a/10370/1424704

candiddevmike 3y ago
> About a fifth to a quarter of public-facing Web servers are Windows Server Got a link for that? Best I can find is 5% of all websites: https://www.netcraft.com/blog/may-2023-web-server-survey/ https://www.netcraft.com/blog/may-2023-web-server-survey/
wg0 3y ago
20% of workloads running on Windows should result in corresponding number of jobs as well but that's not what I see. Most companies are writing software with software developed on Linux first and for Linux first (or Unix) and later ported to Windows as an after thought. I'm thinking Python, Ruby, NodeJS, Rust, Go, Java, PHP but not seeing as much of C#/ASP.NET which should at least be 20% of the market? Only two explanations - either I am in a social bubble so don't have exposure or writing software for Windows is so much easy that it takes five times less engineering muscle.
- argiopetech 3y ago
  There are plenty of .NET jobs, and .NET (Core, particularly) is really easy to write. That said, I'd guess the difference is that the startup and big tech world (i.e., "software companies") like our fancy stacks, but non-software companies prefer stability and familiarity. It makes way more sense for most companies to have a 3-man "bespoke software" department (sys/db admin, sr engineer, jr engineer) on a stack supported by a big company (Microsoft) where most of the work is maintenance and the position lasts an entire career. It's a big enough team to support most small to middling businesses, but not so big that the push to rewrite everything in [language/framework of the week] gains traction. The practical conclusion is that these companies have few spots to fill, and they probably don't advertise where you're looking.

deleted 3y ago

[deleted]

wegfawefgawefg 3y ago

Unix kinda still does the same thing now as before. Future big ai models might be totally different in quality, and latency.

_jab 3y ago

The argument may be that having very large models that everyone uses is a bad idea, and that companies and even individuals should instead be empowered to create their own smaller models, trained on data they trust. This will only become more feasible as the technology progresses.

danaris 3y ago

> But how that can be possible for an LLM? Well, it seems to me that's part of the problem here. And it's their problem, one they created for themselves by just assuming they could safely take absolutely every bit of data they could get their hands on to train their models.

blovescoffee 3y ago

People thinking to dismiss this should, period. Consider that Open AI and similar companies are the only ones in the AI space with the market cap to build out profitable hardware projects which open source can't. Or maybe every investor is just dumb and likes throwing millions of dollars away so they can participate in a hype train.

bigbillheck 3y ago

> But how that can be possible for an LLM? They should have thought of that before they went ahead and trained on whatever they could get. Image models are going to have similar problems, even if they win on copyright there's still CSAM in there: https://www.theregister.com/2023/12/20/csam_laion_dataset/ https://www.theregister.com/2023/12/20/csam_laion_dataset/

015a 3y ago

Windows and MacOS (and their closed source derivatives) are probably at least as large as Linux, even including all the servers Linux is deployed on. Proprietary UNIX did not "die out"; Apple sells about a quarter million of them every year. The majority of the world's computing systems runs on closed source software. Believing the opposite is bubble-thinking. Its not just Windows/MacOS. Most Android distros are not actually open source. Power control systems. Traffic control systems. Networking hardware. Even the underlying operating systems which power the VMs you run on AWS that are technically open source. The billions of little computers that form together to make the modern world work; they're mostly closed source.

benrow 3y ago

Google have their "Machine Unlearning" challenge to address this specific issue - removing the influence of given training data without retraining from scratch. Seems like a hard problem. https://blog.research.google/2023/06/announcing-first-machine-unlearning.html?m=1 https://blog.research.google/2023/06/announcing-first-machin...