9 ms·
Smart Algorithms beat Hardware Acceleration for Large-Scale Deep Learning
- kevingadd 7y agoFor reference since it's kind of buried/obfuscated: Their point of comparison is between an NVIDIA Tesla V100 and a 44-core CPU. The latter is probably something like a Xeon E5-2699, which has a list price of $4115 USD. Hard to find accurate pricing data for the V100, but it looks like it was in the $7-10k USD range back in 2018. Still a cool cost/performance improvement but not as massive as I was expecting before I looked up the test hardware.
- tyingq 7y agoThat's fair, though it's much easier and cheaper to rent CPU time because of the overwhelmingly larger amount of supply.
- lawl 7y agoIt's not just that the CPU is cheaper than the GPU. They also claim it's faster than the more expensive GPU. Meaning if you rent a box in the cloud for training it'll cost you less since you'll rent it for less time, and if you buy the hardware you can train more models and thus need less hardware to train as much as with GPUs before. So if it's 3.5 times faster then you require only roughly 30% of the time (or hardware) it took before, both savings combined seem pretty significant to me. Edit: quick napkin math, let's take your $7k figure for the GPU and 4115 for the CPU. We need one third of that CPU so 4115 * 0.3 = 1234.5 now 1234.5 / 7000 = approx. 0.176. 18% of what it cost before...
- raghavtoshniwal 7y agoAlso worth noting that that Xeon processor takes somewhere around 350-400W and the V100 is also in the range of 300W. So not huge on energy savings. Although a really cool, potentially industry shaking breakthrough, regardless.
- mkl 7y agoThe V100 would need to run for 3.5 times as long, so would use way more energy. E.g. training for 1 hour on CPU: 400W×1h = 400Wh of energy, vs training for 3.5h on GPU: 300W×3.5h = 1050Wh of energy.
- steerablesafe 7y agoWhy is this downvoted? This is absolutely right.
- tyingq 7y agoI realize TDP is a bit gamed, but Intel shows the E5-2699A as 145W. Curious where you got the 350-400W for the CPU. Nvidea lists 300W as the TDP for the V100.
- ahoka 7y agoIntel TDP is not "gamed", but total bullshit: https://www.tomshardware.com/reviews/intel-xeon-e5-2600-v4-broadwell-ep,4514-8.html https://www.tomshardware.com/reviews/intel-xeon-e5-2600-v4-b...
- qayxc 7y agoSince the article doesn't specify how power draw was measured, I assume that they measured _total system_ power consumption. This is not to be confused with the CPU power requirements as it includes the entire platform and all losses (e.g. power supply, main board, peripherals, cooling, ...).
- veselin 7y agoThe new AMD EPYC 7742 should have double the memory bandwidth (they claim they are memory bound) and 1.5x the cores for a similar list price. But in all cases, I would not expect that a serious research can outright beat the current NNs by 10x on all dimensions. This will take some time (and may not fully happen) and this paper is certainly a great advance.
- zvrba 7y ago> Their point of comparison is between an NVIDIA Tesla V100 and a 44-core CPU. Not quite. Further in the paper: "Similarly, for larger Amazon dataset, the red and black line never intersect, and the red and blue line intersects on 8 cores. This means that SLIDE beats Tensorflow-GPU with as few as 8 CPU cores and Tensorflow-CPU with as few as 2 CPU cores." Yes, they have a beast machine, but it can outperform the GPU with only 8 cores in some cases.
- threeseed 7y agoYou can get an Intel 10980xe with 18 cores/36 threads for $1030USD which also supports AVX512 and DL-Boost. https://ark.intel.com/content/www/us/en/ark/products/198017/intel-core-i9-10980xe-extreme-edition-processor-24-75m-cache-3-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/198017/... Availability is pretty much non-existent though. Also you can OC it pretty close to 5GHz with decent water cooling.
- poorman 7y agoMaybe the real benefit here is that due to the asynchronous nature of the algorithm, you could easily scale to more cores and the new approach will scale with it, vs. at some point, you reach a point of diminishing returns with the GPU?
- tyingq 7y agoThe best laypersons's summary I could find: "SLIDE doesn’t need GPUs because it takes a fundamentally different approach to deep learning. The standard “back-propagation” training technique for deep neural networks requires matrix multiplication, an ideal workload for GPUs. With SLIDE, Shrivastava, Chen and Medini turned neural network training into a search problem that could instead be solved with hash tables" https://insidehpc.com/2020/03/slide-algorithm-for-training-deep-neural-nets-faster-on-cpus-than-gpus/ https://insidehpc.com/2020/03/slide-algorithm-for-training-d...
- ganzuul 7y agoSeems this is related to an adaptive approach which GPUs don't have support for, but could be made to. I think this means the next version of TPUs will support it, and then GPUs follow closely after.
- tyingq 7y agoMaybe..."Our analysis suggests that SLIDE is a memory-bound application"
- yvdriess 7y agoNo, their approach changes the fundamental access pattern into something anathema to GPU and TPU architectures. In ELI5 or layman's terms: current GPU/TPU accelerators are specialized in doing very regular and predictable calculations very fast. In deep learning a lot of those predictable calculations are not needed, like multiplying with zero. This approach leverages that and only does the minimal necessary calculations, but that makes it very irregular and unpredictable. Regular CPUs are better suited for those kind of irregular calculations, because most other general software is that as well.
- numpad0 7y agoIn layman's response that sounds like that network could use normalization
- RawChicken 7y agoSorry for the off topic comment but this then/than mistake I read every day is just getting on my nerves. " What to Know: Than and then are different words. Than is used in comparisons as a conjunction, as in "she is younger than I am," and as a preposition, "he is taller than me." Then indicates time. It is used as an adverb, "I lived in Idaho then," noun, "we'll have to wait until then," and adjective, "the then governor."" [1] [1] https://www.merriam-webster.com/words-at-play/when-to-use-then-and-than https://www.merriam-webster.com/words-at-play/when-to-use-th...
- flohofwoe 7y agoFor a non-native speaker it's hard to tell the two apart. Then and than sound pretty much the same. And even if one knows the difference, it's easy to make a typo. In the sort of "international pidgin English" that's spoken anywhere outside the UK such subtle differences should just be ignored.
- ur-whale 7y ago>should just be ignored. As a matter of fact, it shouldn't: the sentence with "then" in it has a different meaning altogether than the one with "than" in it. Using "then" suggests that something is done on the CPU and then ("then") on the GPU.
- dkersten 7y agoIf they are just ignored, how will anyone learn? If they really were small (to the meaning of the sentence) differences, then whatever, but switching than for then changes the sentence.
- heero 7y agoI don't agree, I'm not a native speaker and probably like many, I leaned english by reading so those two words sound very diferently in my head. I'm always lost when I see this mistake.
- Majestic121 7y agoI don't know about that, this kind of mistakes (then/than, effect/affect) are getting on my nerves as well, and I'm definitely not a native english speaker. Not to mention that this specific kind of mistakes (similar sounds) are at least as often from native speakers as from non-native in my experience/native language. In French, a lot of people mistake Ça for Sa for example, native and non-native alike
- bitL 7y agoDid they just accidentally kill NVidia's business model?
- numpad0 7y agoOnly if it works and scales? NVIDIA can still build like 128 core ARM if they absolutely has to.
- zozbot234 7y agoI don't think so. Matrix multiplication has plenty of uses besides neural network training, and GPUs will still excel at those workloads.
- bitL 7y agoMatrix multiplication with INT8, INT16 or BFLOAT16 doesn't have that many uses outside Deep Learning.
- jmmcd 7y agoEven if they killed it, it wasn't accidental!
- qayxc 7y agoI don't think so. Keep in mind that DL training is only a minuscule part of GPGPU applications. HPC is still a huge market with a strong demand for compute accelerator cards. In practice, TPUs and SoCs with inference extensions are a much bigger threat for NVIDIA in the cloud and automotive business.
- imtringued 7y agoNVidia could add special accelerator instructions to their GPUs to efficiently implement the same algorithms and then they would be significantly ahead of Intel and AMD.
- mrb 7y agoAs someone who ported memory-bound workloads to GPU, I say SLIDE appears it would run even faster on GPU than on CPU. I skimmed the paper and SLIDE is described as a memory-bound workload, specifically random memory accesses to 2-10GB of data. GPUs excel at this type of workload. For example the Ethereum PoW (ethash) is memory-bound, and GPU ethash implementations are faster than CPU ones. So I don't understand why the authors don't mention the possibility of implementing SLIDE on GPU. Of course, I could have missed something (I spent less than 10min reading the paper)...
- yvdriess 7y agoNo, that is definitely not the case, not all memory-bounds are of the same type. Hash tables produce lots of data-dependent random accesses into DRAM which are definitely not better on GPUs. Warps divergence, bank conflicts, partial cache-line access inefficiencies, etc. Even CPUs struggle on this type of pointer chasing workloads due to cache inefficiencies. Open addressing schemes such as robin hood hashmaps are popular because they reduce the amount of pointer chasing. Your example is a false comparison, generating a hash is very different from pointer chasing the address generated by the hash.
- pcwalton 7y agoWell, depends on the size of the hash table and the particular memory access patterns. Lookups into many small hash tables, or workloads in which most threads in a threadgroup all fetch the same entry, can be very efficient on GPUs. Sparse virtual texturing is often implemented with a hash table and works well on GPUs because the hash table involved has both of these properties. (I'm sure you know this, just wanted to clarify.)
- yvdriess 7y agoYes, a very good point. I am assuming the tables are quite large due to the workload. If it's large enough to give a benefit to large pages in reduced DTLB misses, it's likely too large for warp-local memory :)
- signa11 7y agoactual paper: https://arxiv.org/abs/1903.03129 https://arxiv.org/abs/1903.03129
- fareesh 7y ago> Slide: 3.5x Faster Deep Learning on CPU Then on GPU Typo in the submission title - it should say "Than" on GPU
- nabla9 7y agoThey don't implement SLIDE in GPU so we don't actually know if CPU is faster than GPU. It's SLIDE on CPU vs softmax & sampled softmax in GPU comparison. They should at least give rationale why GPU is not speeding up locality sensitive hash based algorithm. GPU's are used to compute fast hashes (they were used in Bitcoin mining once). It's Intel sponsored research, but come on.
- yvdriess 7y agoSince when is calculating the hash the bottleneck in hash table access? The bottleneck is the: ptr = h(x); // trivial, ~0.5-2 cycles bucket = *ptr; // --> ~200-300 cycles edit: reading the paper, it's pointers to the data being stored, so I have to add the following as well: el_ptr = bucket[i] el = *el_ptr So that's two dependent random-access loads.
- johnlorentzson 7y agoForgive me if this should be obvious, but why would a simple read from a pointer take so many cycles?
- erikmolin 7y agobecause you have to fetch it from RAM, unless the problem is small enough to fit in cache
- signa11 7y ago> because you have to fetch it from RAM, unless the problem is small enough to fit in cache ... might have to...
- johnlorentzson 7y agoAh, right. I don't know how I forgot about cache and RAM.
- yvdriess 7y ago
- ourlordcaffeine 7y ago>We provide codes and scripts for reproducibility Where? I want to get my hands on this code
- garybake 7y agopaperswithcode has a link to the repo https://paperswithcode.com/paper/slide-in-defense-of-smart-algorithms-over https://paperswithcode.com/paper/slide-in-defense-of-smart-a...
- regularfry 7y agoI think I'm right in saying that the type of locality-sensitive hash systems they're talking about are not entirely dissimilar to Igor Aleksander's WISARD, RAM-based recognisers from the 80's. I suspect the latter is a special case of the former. How far off-base am I?
- LargoLasskhyfv 7y agoWhy do you all guess? The paper has a link to their github with the source and instructions: https://github.com/keroro824/HashingDeepLearning https://github.com/keroro824/HashingDeepLearning I see no reason not to try it with some AMD Threadripper or EPYC instead.
- threeseed 7y agoIt looks like it benefits from AVX512 which AMD does not support. Might be worth trying on something like an 10940x/10980xe if you can get your hands on them.
- thesz 7y agoAVX512 benefits are coming from gather-scatter instructions, I think. What is interesting here is that in their current implementation they aren't very beneficial [1] and [2]. [1] https://arxiv.org/pdf/1806.05713.pdf https://arxiv.org/pdf/1806.05713.pdf [2] https://www.sciencedirect.com/topics/computer-science/scatter-instruction https://www.sciencedirect.com/topics/computer-science/scatte... (recommends these instructions to be used outside of main loop) I remember vaguely that first implementations of scatter/gather instructions were not faster than sequential access from different memory registers. And, thusly, it may come handly that AMD has much bigger core count because each thread will have less memory to access.
- comicjk 7y agoImportant to note that the datasets used are sparse, and that the key to this algorithm is better exploitation of sparsity. The GPU over CPU advantage is a lot lower if you need sparse operations, even with conventional algorithms. "It should be noted that these datasets are very sparse, e.g., Delicious dataset has only 75 non-zeros on an average for input fea- tures, and hence the advantage of GPU over CPU is not always noticeable." In other words, they got a good speedup on their problem, but it might not apply to your problem.
- xiphias2 7y agoI believe that it's so critical here that the dataset is sparse, that it should be in the title of the paper. Like this I view it as clickbait.
- wdobbels 7y agoIt's not even mentioned in the abstract.
- rajesh-s 7y agoAnother thing to note is that sparsity is being leveraged even to build a more efficient version of hardware. A good example of this is the Cerebras Waferscale chip that was announced recently. I'm assuming the author was unaware of developments on the hardware side of things.
- thesz 7y agoWaveNet, if I remember correctly, has 1-from-256 encoding of input features. And 1-from-256 encoding of output features. It is extremely sparse. If you look at language modeling, then things there are even sparsier - typical neural language model has 1-from-several-hundredths-of-thousands for full language (for Russian, for example, it is in range of 700K..1.2M words and it is much worse for Finnish and German) and 1-from-couple-of-tens-of-thousands for byte pair encoded language (most languages have encoding that reduced token count to about 16K distinct tokens, see [1] for such an example). [1] https://bellard.org/nncp/ https://bellard.org/nncp/ The image classification task also has sparcity at the output and, if you implement it as RNN, a sparsity at input (1-from-256 encoding of intensities). Heck, you can engineer you features to be sparse if you want to. I also think that this paper is an example of "if you do not compute you do not have to pay for it", just like in GNU grep case [2]. [2] https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug... Given all that I think it is a paper about combination of very clever things which give excellent results in a synergy.
- tkyjonathan 7y agoThis is obvious to me. Since Hadoop came out, (a lot of) people have been giving up on even forming algorithms and just dumping data into machine learning and hoping for the best. I recall someone high up at Google complaining about it. We need to get back to forming algorithms as well as concepts and first principles. We cannot and should not expect ML to brute force finding patterns and just sit back and relax. Here is another prediction for you: we will not solve ray-tracing in games and movie CGI with more hardware. We will need some algorithm that gets us 80-90% of the way there in a smart way.
- taneq 7y agoThis was my first thought. Well, to be more complete - smart algorithms beat dumb algorithms even if the dumb algorithms use hardware acceleration (unless the problem is trivial anyway.) Smart algorithms plus hardware acceleration beats smart algorithms on general purpose hardware. Smart algorithms are just better.
- bmh 7y agoI'm curious to see if they can apply this to industry-standard tests like ImageNet classification. The workloads that they test on make it hard to quantify the broad applicability of their work.
- andrewmatte 7y agoA reminder for us all: No matter how much faster your hardware, you can still write slower code
- poorman 7y agoI've been trying to get through this paper for the last two days. It's somewhat sparse itself. Maybe I need to go read the code they wrote first...
- mwexler 7y ago"But... Moore's Law, more hardware!" he plaintively cries out...
- m0zg 7y agoA more correct headline would be: "An algorithm beats a poorly optimized GPU implementation on a narrow problem that uses an extremely sparse dataset using $8.5K worth of CPU". This is for recommenders only, and it does not translate to anything else. I don't know why everyone seems to misrepresent this as NVIDIA's undoing. Read the freaking paper, people.
- jandrewrogers 7y ago> "hashing is a data-indexing method invented for internet search in the 1990s" Eh? It was invented multiple decades prior to the 1990s. Some days you could get the impression that computer science did not exist before the Web.
- deleted 7y ago[deleted]
- ironfootnz 7y agoI feel like this is more a stunt for the new intel xenon. Creating a new paradigm on a heterogeneous hardware dependency is misleading. Linear analysis could be more explored. They could achieve the same with less. Holomorphic functional analysis. But it’s a good try. I’d say that this catch’s my interesting as is a valid point to optimizations https://arxiv.org/pdf/1908.05858.pdf https://arxiv.org/pdf/1908.05858.pdf