6 ms·
> Deep learning enthusiasts probably will stick to NVIDIA cards We've been building the wrong hardware for ML for a while now. A TPU doesn't delivery supremac
by doctorpangloss 8y ago
> Deep learning enthusiasts probably will stick to NVIDIA cards
We've been building the wrong hardware for ML for a while now.
A TPU doesn't delivery supremacy for problems over GPU hardware. It's a Google senior engineering retention and PR project.
Exceedingly few problems resemble image recognition. You wouldn't be able to tell from the research, because due to the tools it's sort of the only affordable thing to do.
Playing Go barely looks like it. Or more accurately, Go doesn't look like most other games. Molecular physics and synthesis barely looks like it. Neural programs/neural Turing machines that matter (converting complex sim code like game rules and physics into neural networks via learning) may never achieve the accuracy or performance/watt necessary to really compete with just making a bunch of CPUs or straight up dedicated hardware.
Word embeddings are a disaster. People keep trying to do innovative stuff with it, and it's 2018 and we've just just recently got entity names from Wikipedia. I think Peter Norvig, having basically only machine translation to point to, is really going to eat crow when non-perceptual AI tasks sort of have nothing to do with stuff Google has developed. But what do I know.
I'm sure someone's going to trot out some obscure TPU or GPU-driven thing. It's been years! Awesome innovations in hardware really do lead to obvious, immediate crazy cool stuff. I'm really talking about the Kinect as an example here, in that it was really hyped and not much came of it. The TPU, and ML-specified hardware targeted to today's problems, is a slower-motion Kinect of our time.
Mining cryptocurrency, the most recent innovation, is the opposite of cool.
I'm confident these tools exist to capitalize on the subsidy from the gaming industry. AMD is wise to not chase ML features.
- nradov 8y agoWhat is the right hardware for ML?
- doctorpangloss 8y agoProbably Larabee. Too early and too expensive for its time.
- dragontamer 8y agoThe Japanese PEZY machine looks incredibly interresting. Too bad it is only used in that one super-computer. Its overall design sits somewhere between a GPU and CPU. Like a GPU, it supports 8-way hyperthreading (GPUs typically support 10. CPUs usually only support 2... maybe 4 for Larabee). Like a GPU, its programmed through an OpenCL-like language (I know you don't like that, but its how it works). However, the overall layout is very CPU-like as well: a "City" is composed of "Villages", which is composed of multiple "Processing Elements". Each PE is scalar like a normal CPU, although 8-way SMT. So its not like a GPU's SIMD or SIMT architecture in this regards. -------- See information here: https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc It really is an interesting design that toes the line between CPU and GPU.
- nradov 8y agoIs PEZY still a going concern after the arrests?
- twtw 8y agoSo that's why Xeon Phi has been so successful! /s Real talk, though. Why do you think larabee is the "right" hardware?
- hajile 8y agoI'd assume because of the extra serial processing power of a CPU. x86 isn't the right choice though. As the chips get smaller, all the x86 cruft starts to become a larger portion of each core. RISCV with 40-ish instructions plus a SIMD extension would probably be ideal. They've already proven that those designs use less space per core than ARM.
- vl 8y agoThe truth is that for many types/sizes of models AVS2 or AVS512 in Xeons is as fast and GPUs.
- 8y ago
- marcosdumay 8y agoThat will certainly depend on the kind of ML you want to run. What is the right hardware for computing?
- wyldfire 8y agoSeems like what you need for inference is usually very different from what you need for training.
- shaklee3 8y agoThey absolutely need tensor/matrix multiply cores. Incremental fp32 improvements aren't impressive.
- walterbell 8y ago> talking about the Kinect as an example here, in that it was really hyped and not much came of it The underlying technology was acquired by Apple and is used on iPhones for FaceID, https://www.theverge.com/circuitbreaker/2017/9/17/16315510/iphone-x-notch-kinect-apple-primesense-microsoft https://www.theverge.com/circuitbreaker/2017/9/17/16315510/i... > PrimeSense was never solely responsible for the technology in Microsoft’s Kinect — as evidenced by the huge improvements Microsoft made to Kinect 2.0 on its own — and it’s also obvious that Apple is doing plenty of new software and processing work on top of this hardware. But the basic idea of the Kinect is unchanged. And now it’s in a tiny notch on the front of a $999 iPhone.
- zozbot123 8y agoMatrix multiplication is the bread-and-butter of ML (both training and inference), for fairly obvious and solid reasons. Luckily for us, it's also very important in all sorts of domains, including graphics, rendering and gaming. That's all there is to it. There's no real "subsidy from the gaming industry", only a cross-subsidy situation in which common problems that are shared across domains can take advantage of the increased scale that comes from sharing the GPU-compute platform. I'm not sure what you mean by "ML features", but AMD/ATI's dismissive attitude towards the ML and GPU-compute communities is looking quite silly by now.
- mda 8y agoI'm not sure what you mean by "ML features", but AMD/ATI's dismissive attitude towards the ML and GPU-compute communities is looking quite silly by now. I disagree, ML computing has tons of competition and its margins will be quickly diminish. I think they are right to focus on gaming.
- lawrenceyan 8y ago> Exceedingly few problems resemble image recognition. It makes it extremely difficult to take any of what you just said seriously knowing that you made this statement.
- doctorpangloss 8y agoI don't know dude, if there weren't such lucrative jobs for academics at places like Google and pharmaceutical research companies, the discourse wouldn't be so narrowly focused on doing blah blah neural networks on huge libraries of images and molecular hashes. People with academic backgrounds would be less afraid to point out how much vaporware machine learning has produced. They'd take stock in that their tenure, or six year PhDs, give them an opportunity to not chase trends so aggressively. They'd shill online less, for some giant faceless corporation like Google. I'm not going to shit on a bunch of great stuff people have made. But the scary thing is that Noam Chomsky is probably right, that AI science hasn't advanced much at all. That we're still in an AI winter where it matters, the science, despite all the money and attention.
- lawrenceyan 8y ago> the discourse wouldn't be so narrowly focused on doing blah blah neural networks on huge libraries of images and molecular hashes. Do you mind clarifying what exactly your issues are with these specific applications of machine learning? Is it that you think there isn't really any meaningful value that can be gained in working within these areas? Or is it just that you want people to spend more time researching a different topic that you think would be more interesting/compelling?
- Certhas 8y agoCare to expand? Prima facie this is a very reasonable stance. Further, image data has a lot of structure not present in other data and it's far from obvious what to do with ML techniques in other contexts. This is coming from a physicist starting to investigate ML tools in my own work.
- 8y ago
- dnautics 8y agoI think we are at a point where the cool hardware will actually be low power inference (deployed on basically the class of models we have now).