12 ms·
CPU-based algorithm trains deep neural nets up to 15 times faster than top GPU
- bigmattystyles 6y agoThere are basically no details in the article or the linked one (inside the article). I'll keep an open mind but this is the sort of article that should make you roll your eyes. (◔_◔)
- freeone3000 6y agoIt's legit. For the network types this works on, it's much less computationally complex. A lot of the networks we use now are based on the assumptions of fast matrix multiplication, so it's not universally applicable -- but there's a ton of networks this does work on, and there's a good indication that we can rephrase a model to solve the same problems in the hashtable space.
- king_magic 6y agoI call BS until a paper & repo is released.
- exegete 6y agohttps://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129 https://github.com/keroro824/HashingDeepLearning https://github.com/keroro824/HashingDeepLearning
- yor1001 6y ago44 Cores, and careful cache management. I'm wondering whether the GPU implementation has been similarly optimized.
- guywhocodes 6y agoOur GPU implementations are a lot more optimized. This is pretty far behind MKL etc for matrix multiplication
- eyegor 6y agoThey presented an approach which was not cache optimized which also showed significant gains. And of course they used a lot of cores, they're comparing against a v100, which is a $7k gpu (maybe $5k if you're lucky).
- rsfern 6y agoIt’s 44 threads (22 cores). They also compare with tensorflow cpu compiled with SIMD instruction sets on the same hardware. What other optimizations would you like to see? I would expect the tensorflow team to already pay pretty close attention to performance in cpu and gpu implementations, not to mention CUDNN and such...
- foerbert 6y agoConsidering the GPU implementation is TensorFlow, I think it's very safe to assume the GPU implementation is the far more optimized one.
- d110af5ccf 6y agoIs there anything preventing the same or a similar algorithmic optimization from being implemented on the GPU though? IIUC, the new algorithm (on the CPU) was compared to an existing algorithm (both on the CPU and GPU).
- taf2 6y agoInteresting the golang version https://github.com/nlpodyssey/goslide https://github.com/nlpodyssey/goslide
- king_magic 6y agoI stand corrected!
- exegete 6y agoPaper: https://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129 Code: https://github.com/keroro824/HashingDeepLearning https://github.com/keroro824/HashingDeepLearning Edit: Here is the new paper referred to https://arxiv.org/abs/2103.10891 https://arxiv.org/abs/2103.10891 Here is the new code: https://github.com/RUSH-LAB/SLIDE https://github.com/RUSH-LAB/SLIDE The paper I linked above was the first paper of which this is supposed to be an improvement.
- guywhocodes 6y agoThis is the initial SLIDE paper, this is supposed to be an improvement on top of SLIDE. Having looked at and run the original implementation I'm a lot less skeptical than the average post here. It was quite fast already and should be considered a toy implemenation.
- djrogers 6y agoWhat do you mean by ‘toy implementation’?
- guywhocodes 6y agoPoC for fun
- gumby 6y agoDid you mean, "should not be considered a toy" or "this original was a toy/PoC and this new paper is much better"?
- guywhocodes 6y ago> this original was a toy/PoC and this new paper is much better This one hundred percent, the original has very few optimizations over a completely naive implementation. It uses MPI and huge pages, that's essentially it.
- foerbert 6y ago
- baybal2 6y agoJust as expected. Turning back to nineties, it's easy to see that almost all ventures into specialty hardware ended up with mainstream consumer CPUs catching up, and swallowing the niche in a few years time as computer science, and logic design advanced. Very few computing tasks came out to be truly brute force demanding as people learned how things really work. Multi-channel audio, and sound effects on CPUs — once thought to be impossible, are now everywher, even smartphone chips. Getting there was very tough though. Very few people can write a software DSP, and accoustically correct mixers with real time performance even today.
- gpderetta 6y agoAKA the Wheel of Reincarnation. I do think that GPUs are here to stay for a long time though.
- mcosta 6y agoYes, but how much until integrated beats separate card in $/performance? Once I bought a separate i387 chip for floating point. Later it was inside the main CPU.
- gameswithgo 6y agoLots of jobs where a GPU is a factor of 1000x faster so...
- wongarsu 6y agoBoth CPU and GPU are in large part limited by power delivery and heat. Putting them both in one chip tends to make both of them worse. I wouldn't count on that reversing any time soon. It is however curious how CPU and GPU become more similar over time, with CPUs getting ever wider SIMD instructions and GPUs becoming better over time with branching code, integer performance, etc.
- gpderetta 6y agoThere is some convergence, but CPUs are still latency optimized and GPUs are throughput optimized which makes for vastly different architectures.
- guywhocodes 6y agoIt's not the first similar claim, there was one that large CPU's can beat GPUs by creating a very sparse matrix using spatial hashing. Preserving the gradient but finding it with vastly less computation by simply not multiplying elements that don't have much or any impact. EDIT: orginial work afaik https://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129
- AndrewKemendo 6y agoAKA work smarter not harder
- cat199 6y agoThis is literally the same research group, and the paper you link to is referenced in the article as being the predecessor 2020 publication to this announcement of a forthcoming 2021 paper expanding on the 2020 one. So yes and no - if you are correct and that paper is indeed the first claim (not up on the literature), this one is also the same first claim, or at least, an extension to it
- mlthoughts2018 6y agoThe paper is from 2019. I think the article title should be updated to reflect that, since this link is nothing but rehashing the original algorithm, despite minor embellished claims about “taking it further.” - https://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129
- g42gregory 6y agoLatest 2021 Code: https://github.com/RUSH-LAB/SLIDE https://github.com/RUSH-LAB/SLIDE
- deleted 6y ago[deleted]
- xaduha 6y agoReminds me of https://en.wikipedia.org/wiki/Cell_(microprocessor) https://en.wikipedia.org/wiki/Cell_(microprocessor) and how that was considered a bad direction for game AI and general game depth, at least some were proclaiming it as such, that it would lead to prettier, but dumber games. Didn't make much sense to me at the time and in general I think number cruncher cores were a boon for AI.
- monocasa 6y agoGame AI, being simple symbolic AI, has really branchy unpredictable instruction streams. The SPEs were really more optimized for GPU like tasks with clean loops where you can gang together a few SIMD lanes and process them all at the same time. All of that being said, I've heard from a few PS3 devs that the simple PPEs didn't really do a great job with that kind of AI code either, due to their extremely simple core designs including their branch predictors. So it didn't end up being a concern and a lot of AI ended up running on the SPEs just because they could be laying around with nothing better to process. At the end of the day, game AI wasn't really held back by a particular console's design.
- baybal2 6y agoI think there is a more direct comparison; Amdahl Trilogy https://en.m.wikipedia.org/wiki/Trilogy_Systems https://en.m.wikipedia.org/wiki/Trilogy_Systems A great case of half of the industry riding into a concrete wall following the trendsetter.
- deleted 6y ago[deleted]
- gumby 6y agoHere's the new paper referred to in the article: https://arxiv.org/abs/2103.10891 https://arxiv.org/abs/2103.10891
- RcouF1uZ4gsC 6y ago> Study co-author Nicholas Meisburger, a Rice undergraduate, said "CPUs are still the most prevalent hardware in computing. The benefits of making them more appealing for AI workloads cannot be understated." I don’t think this statement means what he thinks it means. I think “cannot be overstated” is what is desired.
- jdeaton 6y ago> "The cost of training is the actual bottleneck in AI," This really isn't true in general. Maybe in select subfields of AI like language model pertaining this is true.
- canjobear 6y agoWhat other bottlenecks do you have in mind?
- RhysU 6y agoResearcher time: Data munging, apples-to-apples model evaluation, identifying and fixing broken assumptions only detectable due to subtle model behavior.
- plaidfuji 6y agoSure, but I assumed he was referring to the biggest computational bottleneck. All of those others are just tooling problems or data quality problems. For DNNs the hours-to-weeks-long training times make it hard to iterate manually or do any kind of rational optimization of architecture and hyper parameters.
- RhysU 6y agoSure, if we just had stationary, perfect data and perfect objective functions it's only computational, operational, monitoring, and maintenance complexity that holds us back.
- trott 6y agoThis is currently #1 on the front page, but I think HN is celebrating the defeat of GPUs prematurely. The architectures used in this paper have just one hidden activation layer. It remains to be seen whether the ideas in SLIDE are applicable to general deep learning tasks and architectures. From the previous paper: We choose the standard fully connected neural network with one hidden layer of size 128.
- ladberg 6y agoYep, I feel like anyone realizes when they first start playing around with NNs in a jupyter notebook that you get faster training on the CPU until you level up to bigger networks.
- freeone3000 6y agoThat's normally due to the cost of copying the data to GPU space. Here, they've reformulated the training algorithm to be a less-computationally-complex problem. It's a huge difference.
- high_byte 6y agoso you think it was only faster because GPU init overhead?
- 0x008 6y agoWait... How can 1 Hidden layer be called DEEP neural network? I thought the whole "deep" thing was about having lots of hidden layers. It's like saying we have a super fast fibonacci algorithm, but we can only show it for n=1.
- trott 6y ago> Wait... How can 1 Hidden layer be called DEEP neural network? Yes. They describe a general algorithm, but show results for shallow nets. So it's unclear if this works well in general.
- sumnuyungi 6y agoThe title is a bit misleading as this algorithm is for feedforward networks and doesn't yet support convolutional layers or any of the SOTA techniques for image classification... which is why GPUs reign supreme for training deep neural nets.
- waheoo 6y ago> doesn't yet Does that mean it can / will?
- sumnuyungi 6y agoThe original paper includes convolutional layer support in their future work & next steps. But it's not a foregone conclusion that the same speedup will occur.
- choppaface 6y agoNeRF is a good example of a network that doesn't have convolutions yet requires a ton of iterations to train. This paper is particularly relevant to wide networks which are important because CPU memory is currently much cheaper than GPU memory (even for FANG researchers!).
- sumnuyungi 6y agoInteresting, I didn't know that NeRF was simply a feedforward network. I hope that this research group can make more headway into training on CPUs, but I also would like to (naively) see less hyperbolic titles. This paper is not just particularly relevant to wide networks - it's only relevant to wide networks.
- gugagore 6y agoI think you mean to say "fully connected" in place of "feed-forward" when trying to draw a distinction with respect to "convolutional".
- ramoz 6y ago
- deleted 6y ago[deleted]
- dragontamer 6y agoHmmm, it's a bit clickbait to make this a cpu vs gpu claim, unless there was a reason they believe that SLIDE wouldn't work on a GPU.
- carlmr 6y agoIf I understand the paper correctly it's because GPUs are good at doing a lot of parallel matrix multiplications, at the cost of copying to and from GPU memory. The CPU has the advantage that it doesn't need to copy, and it's more flexible, but not as parallel, so if you use caching and limit the parallelism of the computations by exploiting sparsity the CPU wins as it does here. So I don't see the clickbaitiness here, it's the central claim of the paper.
- dragontamer 6y agoIf you are training on the GPU, the bulk of the weight data never leaves the GPU in the first place. It's a fundamentally different programming style, which makes comparisons extremely difficult. Furthermore: you hide latency on the GPU by running more and more parallel instances. Yeah, there is a 5 to 10 microsecond delay in all CPU to GPU comms and back, but just double, triple, or 100x fold the parallelism and batch up more tasks (aka Gustafson's law). Don't run one SLIDE. Run 100x of them in parallel to hide the latency. AMD Vega 64 has 16384 hardware threads (one instruction every 4 clock ticks across 4096 hardware SIMD cores) with up to 10 occupancy. That's 163840 max hardware SIMD threads of conceptual execution. (Occupancy is similar to hyperthreads/SMT). In practice, you run out of VRAM before running out of threads typically. 1000x instances uses 1000x the RAM. --------- Anyway, I've seen an entire generation of programmers try to make CPU only algorithms and only fail year after year. The cryptocoin community would generally prefer CPU mining rather than GPU or ASIC mining. But time and time again, the algorithms adapt and someone is brilliant enough to make the new alleged CPU algorithm work really well on a GPU.
- stjohnswarts 6y agoIt's hard to compare in general but it's not hard to compare when you can take the same problem and run it in the "blackbox" of each system and compare performance. That's actually really easy. Now whether you can say "in general" would require much more varied comparisons than just a couple in an academic paper. Most people don't care what's in the black box.
- The_rationalist 6y agoIs it faster than deepspeed zero 3?
- The_rationalist 6y agoif you aren't fixated on matrix multiplications does this means it is only faster for networks up to a size? If so what is the estimated threshold?
- abakus 6y agoI hope this or related technology won't be used for cryptomining.
- rbanffy 6y agoIf it does, we’ll have a lot of surplus GPU hardware available to run our number crunching. Or room heating
- nl 6y agoNote that this is an alogirthmic improvement, and the same algorithm wasn't implemented on the GPU version they are comparing it to. I don't see any particular reason why we wouldn't see a similar speedup with the GPU version.
- deleted 6y ago[deleted]
- Random_browser 6y agoSeems to work well for sparse data, I like these little clever tricks that yield big impacts.
- peter_d_sherman 6y ago>"Deep neural networks (DNN) are a powerful form of artificial intelligence that can outperform humans at some tasks. DNN training is typically a series of matrix multiplication operations, an ideal workload for graphics processing units (GPUs), which cost about three times more than general purpose central processing units (CPUs). "The whole industry is fixated on one kind of improvement—faster matrix multiplications," Shrivastava said. "Everyone is looking at specialized hardware and architectures to push matrix multiplication. People are now even talking about having specialized hardware-software stacks for specific kinds of deep learning. Instead of taking an expensive algorithm and throwing the whole world of system optimization at it, I'm saying, 'Let's revisit the algorithm.'" Shrivastava's lab did that in 2019, recasting DNN training as a search problem that could be solved with hash tables. Their "sub-linear deep learning engine" (SLIDE) is specifically designed to run on commodity CPUs, and Shrivastava and collaborators from Intel showed it could outperform GPU-based training when they unveiled it at MLSys 2020." PDS: Quote: "All programming is an exercise in caching." - Terje Mathisen
- hebetude 6y agoThis post has Intel Inside.
- satellite2 6y agoIt seems that Intel new CEO is bringing his connections with ycombinator troll farms.
- nestorD 6y agoOne big caveat is that they use a parallel algorithm on a 44 cores machine to compete with a single GPU. I can train a neural network on the GPU in my laptop but it has only 8 cores.
- justAnIdea 6y agoIt all depends on economics, what you have now doesn't really matter a couple years down the line when the simple progress of time will show if this research has been used and built upon or not. 44+ cores, that's already available for workstations, and with the 4th generation of Threadripper (that's supposed to be shipping this year), that amount of cores is probably going to be even cheaper than it is now. On the other hand, GPUs are absurdly expensive right now as a result of multiple factors (people being home and gaming due to covid, crypto miners, lower supply of parts due to covid), but we'll see in a year. Also, I'm pretty sure the goal of this research was not that you can train a DNN on your laptop, but that there is more democratization in the distribution of training power. Maybe some day a mid sized organization will be able to train a competitive model on a buch of off the shelf (or rented) servers...
- zEjNkoy 6y ago> The title is a bit misleading as this algorithm is for feedforward networks and doesn't yet support convolutional layers or any of the SOTA techniques for image classification... which is why GPUs reign supreme for training deep neural nets. > The title is a bit misleading as this algorithm is for feedforward networks and doesn't yet support convolutional layers or any of the SOTA techniques for image classification... which is why GPUs reign supreme for training deep neural nets. d0Hnep0BSgnvG
- zEjNkoy 6y agoKju8Vfxe
- zEjNkoy 6y ago> This is currently #1 on the front page, but I think HN is celebrating the defeat of GPUs prematurely. The architectures used in this paper have just one hidden activation layer. It remains to be seen whether the ideas in SLIDE are applicable to general deep learning tasks and architectures. > From the previous paper: > We choose the standard fully connected neural network with one hidden layer of size 128.
- zEjNkoy 6y ago> What do you mean by ‘toy implementation’? > What do you mean by ‘toy implementation’? > What do you mean by ‘toy implementation’? gsag3aSCdYIu6F
- unnouinceput 6y agoIf an algorithm runs faster on CPU than GPU, then it's only a matter of time before it gets optimized for GPU and at the end of the day will still run faster on GPU.
- HelloNurse 6y agoAt the very least, generic types of parallel training are applicable to these models (e.g. an ensemble of models that see batches in a different order or are initialized differently).
- villgax 6y agoDang, title is misleading. Should be shallow neural networks
- polyrand 6y agoThis reminds me of this paper that also caught my attention. They used an array language to make convolutions run very fast on CPUs. Array Languages Make Neural Networks Fast: https://arxiv.org/abs/1912.05234 https://arxiv.org/abs/1912.05234
- zepmck 6y agoI think this work is quite ridiculous. As far as I could see, they trained the net with a batch size equal to 1.
- novaRom 6y agoELI5: 1) any input vector can be represented by a "similar" another vector, e.g. [.4, -.3, .1] => [+1, -1, 0] INIT: 2) generate a set of random input vectors using only {+1, 0, -1} numbers 3) for every neuron: compute a set of activations for every random input vector 4) combine those sets producing a large hash table where keys are random vectors and values are indices of active neurons FEED-FORWARD: 5) for every input vector, find its nearest neighbor, e.g.: [.4, -.3, .1] => [+1, -1, 0] 6) use this vector as hash key to get indices of neurons 7) compute activations for only these neurons using only their weights and the input vector 8) repeat for all layers etc. This way, you avoid lots of computations, but the key is to have a good hash function and a lot of memory.
- 29athrowaway 6y agoJust a reminder that biological networks do not do backpropagation. There's a lot yet to learn about neural networks and the state of the art is still biological evolution.
- ilaksh 6y agoI wonder if this is somehow similar to Reformers.
- Roark66 6y agoMy understanding is that this works better because it is faster to hash their data (and compare with prehashed weights) rather than multiply the data and calculate activation functions. I imagine this could work for more layers easily,and using other than fully connected layers could be done by partitioning data (for each set of inputs). The next obvious question is, can their hashing function run even faster on a GPU? They say: >In particular, SLIDE uses recently proposed DTWA hashing which works nicely on sparse data. If we represent the data vector as a list of indices and values. DTWA operation computes a random hash map of every non-zero index in the list. Then the map is partitioned into a fixed number of bins. Finally, the index with the maximum coordinate value in each bin is the hash value. What is the "coordinate value"? And how does the above map to find instances where data and weights values would result in a value above the activation threshold? They themselves mention it "works well on sparse data". If they detect irrelevant data by simple value comparison(wrapped in that hashing function) couldn't GPU algorithm be sped up by skipping those multiplications too? How about instead of running matrix multiplication on entire data and all weights we "preprocess" data and weights finding specific indices where they are both above a certain value and multiply just those. It would be interesting to try that. How expensive would running such comparison, then data copy (or create a map of indices to process), then process only those chosen be in comparison with multiplying everything every time? I'm guessing that it really depends on what percentage of data is actually relevant. Another idea to make GPU AI faster is improving scheduling of those matrix multiplication. Multiplication by zero, another small number or a power of 2 should be able to execute faster than multiplication of high values (assuming it is implemented in hardware by addition and bit shifting if multiplying by powers of 2). I don't know if current GPU algorithms make any use of that, or if they simply divide the data by number of cores and run all chunks in parallel for however long the longest chunk takes to complete? I think GPUs are pretty far from being dead in AI as a result of this.
- Ambix 6y agoInteresting. As I know, Stockfish [1] chess engine exploits NNUE nets optimized for CPU and regularly beats up Leela [2], which explots traditional GPU-based networks based on AlphaZero architecture. [1] https://github.com/official-stockfish/Stockfish https://github.com/official-stockfish/Stockfish [2] https://github.com/LeelaChessZero/lc0 https://github.com/LeelaChessZero/lc0
- Prof-Anshu 6y agohttps://intel-hpc-ai-pavilion.gallery.video/detail/videos/hpc/video/6164686921001/slide-a-sub-linear-deep-learning-algorithm-that-does-not-need-a-gpu?autoStart=true&page=1 https://intel-hpc-ai-pavilion.gallery.video/detail/videos/hp...