7 ms·
AI training is outpacing Moore’s Law
- visarga 5y agoThese large models are going to be the operating system of the future. When they are going to run efficiently on edge we are going to see an explosion of automation. Being so large they are not only more accurate, but also quicker learners. So less initial effort to start a project. Google is teasing us with leaks about Pathways[1], but not releasing a paper or demo yet. [1] https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://blog.google/technology/ai/introducing-pathways-next-... I call them multi-multi-models: multi-modality, multi-task, multi-language, multi-memory (besides the processing memory it can have search engine in the loop and episodic memory). Somebody's going to make it in the next couple of years.
- Moru 5y agoThe sad part is that we are all helping to train the futures AI but who will own it and make money on it?
- wetpaws 5y agoYou can have ownership in companies that do machine learning. It's called "shares". Creating your own business is an option too.
- frenchyatwork 5y agoThe complaint was that it's the people who own the equipment (or shares in it), and not the people who do the work, who are reaping the profit. When the working class had this problem a couple centuries ago, it was solved by building guillotines. I hope we can do better this time.
- thehappypm 5y agoYou’re conflating two things. Feudalism is like slavery: taking credit for someone else’s labor. Automation removes the need for labor. Automation has almost universally led to quality of life improvements, not revolution.
- rowanG077 5y agoTo start an AI company you generally don't need much. A few thousands in GPUs and you are good to go. If AI researchers feel they are underpaid there is literally nothing stopping them from starting a company themselves.
- gmadsen 5y agothat is absolutely not true. I'm not sure where you are getting your information, but read anything from an angel investor. most AI start ups fail because they go into huge debt getting correct data and training models, which end up bankrupting them in the process with nothing to show for it
- rowanG077 5y agoIf you need to collect date then you are not only an AI company. And in that case a large company is indeed needed.
- fartattack 5y agoSorry, are you claiming that the French Revolution occurred due to bad labor practices? Or did trust busting and unionization in the late 19th century US utilize guillotines? Either of those would certainly be a.. uh.. let's go with "novel" retelling of the history! Or did you mean Red October? Because I'm pretty sure /that/ revolution didn't actually solve any problems for the worker in the way things actually panned out
- wetpaws 5y agoI don't see any problem with it, stock market is open for everyone.
- visarga 5y ago> who will own it and make money on it? The nice part about it is that everyone will have access to great models for direct use, fine-tuning and adaptation. But if people don't take this freedom and are content with simply consuming applications, then they depend on those who deploy the apps.
- andreyk 5y agoIMHO (as a AI researcher) 'multi-modality, multi-task, multi-language, multi-memory' may take longer than you think. Doing multi modal learning is quite hard due to lack of labeled data, a lot of tasks either don't offer clean plentiful datasets or require RL / human in the loop learning and not just supervised or self supervised learning, and on and on. Certainly a lot of progress will be made and partial automation is likely to speed up (as has happened with Copilot), but full automation seems to be a good while away. Plug - I expand on this a bit in this blog post 'The Inherent Limitations of GPT-3' (which was on front page of HN a few weeks back and led to a good deal of discussion ) https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3 https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3
- oblio 5y agoI guess the next step is democratizing ML. Right now we seem to be into a kind of "mainframe phase", where running big models is super expensive. I wonder what kind of speedup we need for ML to be truly accessible for the average developer. 50x? 100x?
- hervature 5y agoDefine "accessible". Very modest hardware (sub $100 and certainly sub $1000) can train random forests very quickly. This is going to be close to SOTA on basically any classification task.
- ska 5y ago> speedup we need for ML to be truly accessible for the average developer. 50x? 100x? The data scale (and access) is often more important than the modelling, and that doesn't have a technological fix, or at least not a purely technological fix.
- fumar 5y agoSo Moore's Law is a constraint for average people as hardware costs a key factor in performance, is that right?
- ska 5y agoWhat I'm saying is the hardware is less of a problem than access to data, for most problems that would get you out of what you can reasonably do on a consumer accessible (cost) machine.
- brrrrrm 5y agoThe article is a bit lacking in details. Here are some changes in ML performance that are happening today and don't require increased transistor counts: 1. newer optimizations algorithms converge faster (Adam, adagrad) 2. lower-precision numerics can be used (TF32 is 19bit, bf16 is popular) 3. matrix multiplications are implemented in hardware (tensor cores, AMX) 4. neural networks are compiled directly (TVM, halide) 5. multi-device network topologies are tuned for training (e.g. nvlink improvements) 6. companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)
- JL-Akrasia 5y agoThe last one is particularly effective.
- wing-_-nuts 5y agoI didn't really understand #6? Expand?
- hailwren 5y agoYou can overfit your data by coming up w/ a model that is very specific to it. Consider an n dimensional data set where one variable is a boolean. A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases. They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments. (This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)
- lettergram 5y ago> companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results) No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon
- Der_Einzige 5y agoI love this because it's vindicating Rich Sutton's essay titled "The Bitter Lesson". http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation." And now we are beating out Moore's law!
- dejj 5y agoSeems like a step towards singularity.
- Jiro 5y agoDo we even still have Moore's Law? All indications are that it died out a while ago.
- e12e 5y agoReally? http://www.transistorcount.com/ http://www.transistorcount.com/ Slope seem a bit more gentle - but certainly not gone?
- hyperpallium2 5y agoYou're right about literal Moores Law, but its benefits are much diminished, of performance, power usage and cost. My understanding is the main problem is heat dissipation. I've long predicted macrochips to solve this, brute force of huge quantities of silicon, but it hasn't really happened.
- rictic 5y agoGraphics cards and neural accelerators have continued an exponential improvement in performance per watt dollar, which is the intuitive thing that people often mean when they say Moore's law.
- adverbly 5y agoIf we hit a trillion transistors by 2030 without increasing chip size, I'll be shocked.
- e12e 5y agoI guess it's inevitable that we'll reach a "hard" limit at some point, with exponential increase in density (electrons and photons have a "size"). But I don't know how that translates to current densities (how many atoms in a gate on the Apple m1 chips, and what is the theoretical minimum?).
- bitshiftfaced 5y agoMoore forecasted a doubling of transistor points but a halving of costs. The costs haven't been keeping on pace. https://qph.fs.quoracdn.net/main-qimg-81fb704374c332ad5c5e27e11cbd4ac5-lq https://qph.fs.quoracdn.net/main-qimg-81fb704374c332ad5c5e27... https://www.quora.com/If-the-Moores-law-is-slowing-down-can-the-slow-down-be-solved-by-increasing-the-size-of-the-chip-so-the-chip-can-have-more-free-space-where-more-transistors-will-be-put-If-this-is-not-a-good-solution-what-other https://www.quora.com/If-the-Moores-law-is-slowing-down-can-...
- armatav 5y agoYup, and we don’t currently have an architecture that can support models 35+ years from now. Not yet at least.
- chx 5y agoHave we considered yet whether this is, to quote Stephen Diel, another handwavy technobabble nothingburger? I have very serious misgivings about all this what we currently call AI. Also, even if it is something, is it beneficial to all? Sure it can be beneficial to some corporates/VC but we didn't stop at the dawn of social media to think on the societal impact and we already know there's a lot of algorithmic bias being cooked here.
- bananaquant 5y agoIf I'm reading the plot from the article right, the most impressive gains have happened in the first half of 2019. Since then, the progress has been notably slower, with year-on-year performance increases for some network architectures falling behind Moore's law already.