6 ms·
Correcting Intel's Deep Learning Benchmark Mistakes
- dmoy 10y agoWell that was a pretty concise explanation of mistakes. Benchmarking is indeed difficult to get right.
- mtgx 10y agoEspecially when you compare on purpose against your competitor's 4 year old product.
- NotQuantum 10y agoShots fired, wow Nvidia is pissed that Intel is trying to edge into the market with their Phi series.
- Gladdyu 10y ago`each with one or two sockets of less-capable processors, like Xeon Phi.` Corporate smacktalk...
- modeless 10y ago> Titan uses four-year-old GPUs ... as does nearly every public cloud provider. I agree with most of the article, but you can't fault Intel for benchmarking the hardware that cloud providers are actually offering. I'm not sure what exactly NVIDIA is doing with their Tesla product line but whatever it is, it's really restricting the availability of recent GPU hardware. Even Azure's GPU instances released this month are using the Kepler architecture from 2012. It's fully two generations out of date now, and that's sad.
- pavanky 10y agoI mean Intel is comparing their publicly unavailable product against NVIDIA's publicly available product. Now NVIDIA is replying with the benchmarks on their (as of yet) publicly unavailable product. I think this blog post is fair game.
- vegabook 10y agoyet we should keep in mind that Intel makes its own chips, while Nvidia clear has an availability problem[1] which it cannot fix on its own. Therefore to the extent that Intel controls its own destiny better, we may find that to be a major factor in adoption rates for large installations. [1] http://semiaccurate.com/2016/08/01/nvidia-finally-shows-off-working-gp100s/ http://semiaccurate.com/2016/08/01/nvidia-finally-shows-off-...
- scottlegrand 10y agoNot even wrong. I have two PCs with 4 Titan X (Maxwell) GPUs and a third PC with 4 Titan X (Pascal) GPUs. Both of these systems are available today (I built them myself, total BOM about $7K), and both will destroy 4 Xeon Phi servers at Deep Learning. The benchmark Intel presented here is as disingenuous as their infamous white paper from 2010: http://pcl.intel-research.net/publications/isca319-lee.pdf http://pcl.intel-research.net/publications/isca319-lee.pdf In comparison, a single Knights Landing Xeon Phi will be ~$7K. I know where I put my money. Caveat Emptor. But Xeon Phi and I go way back here. They've been trying to beat my AMBER GPU code since 2013 or so. Many man years later I believe that a Knight's Corner is now ~35% faster than 2 Xeon CPUs with 1M atoms or more (source: http://adsabs.harvard.edu/abs/2016CoPhC.201...95N http://adsabs.harvard.edu/abs/2016CoPhC.201...95N) Meanwhile, the CUDA code has continued to scale with the GPU roadmap and a Titan XP is arguably 9-10x faster than 2 Xeon CPUs. No data is supplied at the low-end for Xeon Phi and I think we can safely assume it's because performance there sucks. (source: http://ambermd.org/gpus/benchmarks.htm http://ambermd.org/gpus/benchmarks.htm) Xeon Phi? IMO avoid avoid avoid until they start winning head to head 3rd party benchmarking fights like Soumith Chintala's fantastic convnet benchmark data: https://github.com/soumith/convnet-benchmarks https://github.com/soumith/convnet-benchmarks
- throwaway287391 10y agoYes to all of this. I'm really surprised they didn't compare costs in this blog post. Ignore the DGX-1 row of their table; the really damning comparison is between the 2nd and 4th rows of the table. With a single 4x GPU server costing around $7k in total (row 4), you get nearly double the performance you get from spending $28k on four Xeon Phi servers (row 2). And that's assuming you've spent the time and disk replicating your data on all four of those Xeon Phi servers, or went to a likely relatively large amount of engineering effort to ensure that network IO doesn't bottleneck training.
- deleted 10y ago[deleted]
- StreamBright 10y agoTrue, however the cloud vendors do not have Intel latest greatest hardware either, so than comparing latest Intel Phi to some older Nvidia GPU does not make sense.
- ChuckMcM 10y agoOk, its officially the new new thing when corporate communications types are sniping at each other :-) Once we start seeing press releases of products that are not available yet being shown to kill the existing competition we'll know that the hype train has officially gone super-product-sonic (that is a hype wave travelling faster than the product releases can support it).
- sliverstorm 10y agoDid you see the Opera vs Edge sniping a month or two ago? I suppose Opera is probably lot less corporate than Intel & Nvidia, of course...
- wmf 10y agoI think both Knights Whatever and DGX-1 are already there. They seem to be in "order now, receive sometime" mode.
- sp332 10y agoXeon Phi is in 23 of the Top500 supercomputer list, so it's not like they're not shipping. The next version, Knight's Landing, should be launching soon, and hopefully will have better availability.
- mtgx 10y agoIntel was comparing Knights Landing to Kepler, which was released a few years back.
- paulmd 10y agoKepler was the default GPGPU product for many tasks until GP100 released - and GP100 still has not hit general availability yet. Essentially, anything that needed dynamic parallelism (launching kernels from within kernels, i.e. tasks where you don't know where the difficult/interesting needles are within a haystack), advanced/concurrent scheduling capabilities, or FP64 is going to be much, much better off with Kepler until users can get their hands on GP100 cards. Maxwell is good at neither of those things - it's actually only good at specifically deep learning/neural nets. Which is not every task within the GPGPU space.
- iamleppert 10y agoWhy don't they provide a link to their testing methodology? They need to back up their claims (on both sides) with the actual configuration, all versions, and sample datasets for people to independently verify. A docker container that runs their performance suite would be ideal.
- onalark 10y agoExcept that Docker containers play terribly with virtualization solutions. Still, some sort of configuration/infrastructure-as-code would go a long way.
- mctx 10y agoWhat about NVIDIA-docker? https://github.com/NVIDIA/nvidia-docker https://github.com/NVIDIA/nvidia-docker
- StreamBright 10y agoIf you would like to measure Docker overhead it would perfect. Unfortunately not all the world is using Docker for everything so running performance tests using the least amount of complexity is usually better for these tests.
- cynod 10y ago@imaleppert Agree getting the details on the testing would be helpful. I think NV was more pointing out that Intel were putting their latest against NV's oldest. It'd be like testing a RX480 against a GTX660. What's the use of that? @modeless the new Azure instances have M60's or you can purchase a 1080 or new TitanX which are both available (although stock has been tight). https://azure.microsoft.com/en-us/blog/azure-n-series-preview-availability/ https://azure.microsoft.com/en-us/blog/azure-n-series-previe...
- modeless 10y agoYeah but Azure is also introducing new K80 instances (based on four-year-old Kepler) becuase the M60 is worse in memory capacity and memory bandwidth, and not much better in FLOPS. NVIDIA is just not putting their best hardware out there for cloud providers to use. Sure I can buy a Titan X for myself (already have), but I can't rent a hundred, or even one, on EC2 or Azure or GCE. And I can't get a P100 yet at all. I don't want to hear NVIDIA claiming unfair benchmarking and citing P100 numbers until P100s are actually available either to buy (and ship immediately) or in the cloud.
- ldargin 10y agoI'm a bit surprised that Nvidia mentioned nothing about performance per watt in their reply.
- NotQuantum 10y agoOr that they didn't mention cost for performance. They equated 4 Xeon Phi servers to ONE DGX-1. The DGX has a 140k price tag.
- taneq 10y agoThat usually implies something pretty specific, like maybe that's the only terms on which they're competitive?
- nzjrs 10y agoI actually came away from the article thinking, heh Phi looks pretty good.
- StreamBright 10y agoI do not usually believe in any vendor provided performance benchmark unless I fabricated it myself. On a more serious note, benchmarking is pretty hard and people usually discount things that seem minor until you found it the bottleneck, like the interconnect in this case for example. Another problem with synthetic benchmarks is that you can always optimize for your exact use case and it usually yields to pretty good improvements, comparable to buying faster equipment. The ultimate quiestion which is more cost efficient, buying faster CPU/GPU or hiring a performance expert.
- sidkashyap 10y agoThey do not mention the version of Caffe used to test the Intel systems, Intel claims its numbers based on an optimized branch of Caffe and not the public (BVLC) version