timdettmers

5 ms·

1. 1.▲
   by timdettmers 6y ago

   Thank you so much!
2. 2.▲
   by timdettmers 6y ago

   Haha, 649 pages! Thanks for the discussion. I can understand your perspective. Maybe it would be best to add something to the blog post that discusses it from both perspectives and readers can then choose which perspective suits them better
3. 3.▲
   by timdettmers 6y ago

   Thank you! You have a good point. I think I would agree with you if somebody already has cloud computing skills, then the cloud is much more powerful to learn deep learning than your own GPU. I figured that most people that start with deep
4. 4.▲
   by timdettmers 6y ago

   I just did a small fix to address this. Thanks again.
5. 5.▲
   by timdettmers 6y ago

   Thank you, I just updated the blog post with more detailed clarification of where the data comes from. One thing that I am quite sure of for the A100 is its transformer performance. It turns out, large transformers are so strongly bottlenec
6. 6.▲
   by timdettmers 6y ago

   I should have been a bit clearer what went into the charts. I do not use theoretical marketing numbers but real-life benchmark data from NVIDIA and 4 other sources of benchmark between Titan V, V100, RTX 2080, RTX 2080 Ti and Titan RTX. Sin
7. 7.▲
   by timdettmers 6y ago

   This is good feedback. Will note this down and incorporate it in a small update.
8. 8.▲
   by timdettmers 6y ago

   What else would you like to see?
9. 9.▲
   by timdettmers 6y ago

   Is up again and stable it seems. I shot my cache while trying to "tune" it. Well, I learned my lesson.
10. 10.▲
    by timdettmers 8y ago

    Please see my detailed answer on this question that you can find on quora: https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learn...
11. 11.▲
    by timdettmers 8y ago

    Your data are inconsistent with the benchmarks that I mention in the blog post: https://github.com/u39kun/deep-learning-benchmark You also do not benchmark LSTMs: https://www.xcelerit.com/computing-benc
12. 12.▲
    by timdettmers 8y ago

    Thanks for your feedback! Someone mentioned this on twitter as well and I thought it was a good point so I implemented that change.
13. 13.▲
    by timdettmers 9y ago

    The site is up again. I have a prepared draft for a blog post exactly on the topic of cloud computing and deep learning. I did not finish it as I thought that there would not be much interest in the overall question since most people will j
14. 14.▲
    by timdettmers 12y ago

    A very valuable comment – this is an important perspective, thanks! I think I will add a EC2 section to my blog post.
15. 15.▲
    by timdettmers 12y ago

    Thanks for sharing your experience – this is a fair point. Often it is possible to pre-process your data and save it to disk so that you can skip this decompression/conversion/transformation step once you start training your net,
16. 16.▲
    by timdettmers 12y ago

    Thanks four your feedback, this is an important point and I will update my blog posts accordingly.
17. 17.▲
    by timdettmers 12y ago

    These are some good points. I heard from another person that he had problems with the form factor and I will add that to the post tomorrow. I think a 2kW PSU is overkill, but you are right that more is better for PSUs. If you want memory a
18. 18.▲
    by timdettmers 12y ago

    If you have no desktop PC or no money for a GPU, it might be a better choice to use a EC2 instance instead of buying the hardware. You pay about $11 a week for a EC2, which is quite good once you compare it against the electricity costs tha
19. 19.▲
    by timdettmers 12y ago

    There are several cryptocurrency mining benchmarks which are bandwidth bound; those benchmarks reflect almost one to one the deep learning performance of a GPU. You can see this for example for litecoin mining: https://litecoin.i
20. 20.▲
    by timdettmers 12y ago

    Thanks for your feedback. I updated the blog post with a small NVIDIA vs. AMD section.
21. 21.▲
    by timdettmers 12y ago

    Thanks for elaborating, these are exactly the points why AMD's GPUs are just not used in deep learning. AWS is great if you want to use a single or two separate GPUs. However, you cannot use them for multi-GPU computation as the virtua
22. 22.▲
    by timdettmers 12y ago

    If you are dissatisfied with the 970 you might be able to get a refund: NVIDIA advertised specs that differ from the real thing – that is a good reason to demand and get a refund.
23. 23.▲
    by timdettmers 12y ago

    Tesla GPUs are also rather slow, but the value of Tesla lies in increased double performance, larger memory and memory correction -- and you cannot beat Tesla GPUs in these departments.
24. 24.▲
    by timdettmers 12y ago

    AMD cards were much faster than the Kepler equivalents, but now NVIDIA fixed their mistakes in that architecture and they are as fast as AMD cards. So AMDs hardware is good, it is just a software issue.
25. 25.▲
    by timdettmers 12y ago

    Why I did not mention AMD cards: NVIDIA's standard libraries made it very easy to establish the first deep learning libraries in CUDA, while there are no such powerful standard libraries for AMD's OpenCL. Right now, there are just
26. 26.▲
    by timdettmers 12y ago

    I mainly use python and sklearn for Kaggle competitions for my initial models. If I understand the problem better I use some of my own deep learning solutions in python (built on gnumpy and cudamat). However, sometimes my own C++/CUDA