3 ms·
Aim wget at The-Eye.eu, figure out how to get hold of complete snapshots of Library Genesis and sci-hub, scrape everything from HN to Stack Overflow to zombo.co
by CamperBob2 3y ago
Aim wget at The-Eye.eu, figure out how to get hold of complete snapshots of Library Genesis and sci-hub, scrape everything from HN to Stack Overflow to zombo.com, and invent a distributed training implementation that we can all sign up for and contribute to, a la Folding@home and SETI@home.
Nothing else will get those patronizing asshats at OpenAI(sic) out of the way of progress. Either we control our models, or they control us.
- flangola7 3y agoYou can't distribute LLM training. It requires high capacity low latency links. Even PCIe is a bottleneck
- CamperBob2 3y agoThat's my point -- there is no more important research problem in computer science right now, IMO, than figuring out how to distribute training. I don't believe that it is provably impossible, or at least I haven't seen any such formal proof.
- flangola7 3y agoHopefully nations will get that on lockdown before it becomes feasible. We do not need the digital equivalent of distributed uranium enrichment.
- CamperBob2 3y agoWell, that's certainly... a point of view.
- MagnumOpus 3y agoOk, that will cost about $100m just for the GPUs, the power, and the data centre bill. How are you proposing to raise that?
- CamperBob2 3y ago$100m is not a problem if a way can be found to allow millions of users to contribute idle resources to the task. Yes, that's like saying "Just invent a practical fusion reactor"... but we're also trying to do that, and this is almost as important.