6 ms·
I'm quite sure even OpenAI themselves aren't sure if they can reproduce the current models from the scratch. Unless the computing becomes much more powerful and
by esjeon 4y ago
I'm quite sure even OpenAI themselves aren't sure if they can reproduce the current models from the scratch. Unless the computing becomes much more powerful and much cheaper, LLM is more or less a rocket science (i.e. hella expensive trial and error). It's not easy to burn lots of dollars just to get what's already there.
- flangola7 4y agoWe have to identify a better method. You can't trial and error a pivotal act.
- mvuksano 4y agoI don't even care if it's reproducible or not. I care it gives me correct responses to my questions and that's all.
- diputsmonro 4y agoThere's really no way to be sure that it will.
- ehnto 4y agoIsn't part of making it reproducible also part of ensuring correct results? Especially if we start putting these models into important systems. And if these models begin to update in an evergreen fashion, or utilize realtime data, getting verifiable or repeatable outputs will be a nightmare if we have no idea how to make these models repeatably.
- visarga 4y ago> if we have no idea how to make these models repeatably We have idea about how to reproduce but using deterministic training mode is very slow as it loses some optimisations.
- pffft8888 4y agoIn this sense, it's more hacking than crareful and well specified engineering, and that could lead down a path of instability in the product where some features get better while others get worse, without understanding exactly why.
- runnerup 4y agoI mean pretty much all real engineering started with that time periods “hacking”/“tinkering” before thorough models and equations were derived. We had 200 years of tinkering with relatively modern steam engine technology before Carnot and Watt started just barely scratching the surface of the first principles of thermodynamics and engine efficiency. Even the eponymous Carnot cycle wasn’t rigorously defined mathematically during Carnot’s life. That being (T1−T2)/T1 as the temperature delta part of the equation, because absolute temperature hadn’t been accepted and defined by Lord Kelvin yet. Some decade later the first law of thermodynamics was finally invented. Hundreds of years of experimentation until the first principles. Machine learning has lots of control systems theory and information theory to help with analysis but we barely have an “engineering” in “software engineering” today, let alone in “machine learning engineering”. We’ll get there, but it’ll be awhile before there are proven design equations with rigorous derivations from first principle that allow us to design and build a precise AI model as surely as we can design and build a precise bridge or levee or distillation column. Let the hacking continue, let’s not worry too much about the future “engineering” that will follow in its own time. Unless you want to discover it yourself or fund its discovery.
- localplume 4y ago[dead]
- ChatGTP 4y agoIt's fine to be hacking, if you're not making billions off the service which people expect some type of stability or baseline performance from, at least that's how I interpret what the parent is saying. Maybe it's easy enough for them to just copy the model, tweak, hack and play with it from there with little interruption. No one really knows at the moment.
- randomwalker 4y agoSure, but the article is talking about a completely different meaning of reproducibility, where a researcher uses an LLM as a tool to study some research question, and someone else comes along and wants to check whether the claims hold up. This doesn't in any way require the training run or the build to be reproducible. It just requires the model, once released through the API, to remain available for a reasonable length of time (and not have the rug pulled with 3 days' notice).
- buildbot 4y agoNot to be impolite, but this is incorrect. One detail they did share in their paper is that they where able to finetune and select their hyper parameters on a model that needed 1,000x less compute than the final gpt4 model. OpenAI is definitely leading in how to train very large models cost effectively.