10 ms·
Infrastructure for Deep Learning
- thr0waway1239 10y agoI don't know much about deep learning. Just noticed that there are 40+ upvotes and 0 comments. I propose the HN Bikeshedding effect theory. Take the number of comments and divide it by the number of upvotes. <0.1 = Too technical for even HN audience 0.1-1.0 = At the right level for the HN audience >1 = The topic is similar to painting the bike shed.
- TheAlchemist 10y agoSounds about right ! But actually all Deep Learning stuff, tutorials, videos are upvoted pretty quickly those times.
- minimaxir 10y agoThe high amount of upvotes and low ratio of votes/comments on deep learning/big data posts is unfortunately accurate. It's not a problem that HN has topics which are frequently upvoted; topics such as employment and Rust are popular memes. It is a problem, however, if the upvote-for-the-title crowd upvote articles which are bad and would not get upvotes if they were about another topic. That's a legit hard problem to solve (what makes a good submission?), unfortunately, but one I've been looking into. (For clarity, this submission is a good submission, but I've seen quite a few top-ranking HN submissions that are just a bar chart on a controversial topic that is poorly sourced. And linkbait about deep leaning tends to get upvotes, but flagged too.)
- thr0waway1239 10y agoI was just making an observation, certainly didn't intend it as criticism.
- mamon 10y agoThere's a problem with upvoting on HN: upvote is also the only saving mechanism available: If I run across some click-baity title, but I don't have time to read it right now I will click upvote, but what I really mean by that is "save for later". Maybe HN just needs to separate bookmarking and upvoting?
- detaro 10y ago... HN has added favorites recently ("favorite" link, on the submission/individual comments page) and thus has both. And apparently really needs a more public changelog.
- asimuvPR 10y agoHave you considered that some posts may simply be a good read? I often don't comment because the post itself made a good job of stating a point. There is no need to reiterate what was said in a post just to get HN points.
- thr0waway1239 10y agoHN is like Leslie Knope and has an opinion about everything :-) Jokes aside, I agree with your comment.
- asimuvPR 10y agoI've created two scrapbooks commemorating this comment. Where I should ship them to? ;-)
- thr0waway1239 10y agoBurt Macklin, FBI.
- Houshalter 10y agoThat's already how HN works. Articles with too many comments get a huge penalty and get pushed off the front page quickly. It has the effect of stifling a lot of interesting and important discussions and topics. I make an effort to up vote posts with too many comments to help them.
- vonnik 10y agoTensorflow is actually pretty slow and problematic on large clusters outside the Google Cloud. Probably because that's not what it was designed for. For Java/Scala people, Deeplearning4j has a pretty sophisticated Spark + GPUs setup: http://deeplearning4j.org/gpu http://deeplearning4j.org/gpu http://deeplearning4j.org/spark http://deeplearning4j.org/spark http://deeplearning4j.org/spark-gpus http://deeplearning4j.org/spark-gpus [Disclosure: I help create DL4J, and it's supported by my startup, Skymind.]
- minimaxir 10y agoYou should disclose that Deeplearning4j is your startup.
- vonnik 10y agoDone!
- rryan 10y ago> Probably because that's not what it was designed for. TF was explicitly designed with distributed training in mind (their initial whitepaper and the DistBelief paper that came before it make this clear) -- I don't know how you came to this conclusion. Usually when people say TF is slow, it turns out they've introduced a serious bottleneck somewhere.
- vonnik 10y agoThe environment for your large clusters is almost certainly different from everyone using TF outside of Google. I'm speaking about the problems they'll run into.
- nl 10y agoI'm speaking about the problems they'll run into. What are these mythical problems you speak of? I'd hear to hear some specifics, because I haven't hit them yet.
- 10y ago
- ymt123 10y agoIt's great to see people talking about the infrastructure they use to manage their deep learning workloads. One area where we've had trouble with other orchestration tools (e.g. Docker Swarm) was in managing resources at anything beyond whole boxes. They are all good at managing CPU/RAM/Disk but we've had trouble with give this task GPU2. We had planned to try Mesos (given that we already run it for other things) but it sounds like maybe we should take a harder look at Kubernetes first.
- programnature 10y agoWhile its useful to have this kind of info, IMHO its still far from 'infrastructure for deep learning'. What about model versioning? What about deployment environments? We need to address the whole lifecycle, not just the 'training' bit. This is a huge and underserved part of the problem bc people tend to be satisfied with having 1 model thats good enough to publish.
- asimuvPR 10y agoSo true. I've been doodling some tools to somehow manage all of it. So far I only have git-like approaches to models and chef-like approaches to infrastructure. I hope to somehow bring all together into a docker-like package that can be deployed without much hassle.
- daveguy 10y agoYou might want to check out Pachyderm -- that is essentially what they are trying to do (Analytics infrastructure support. It isn't specific to machine learning): http://www.pachyderm.io/ http://www.pachyderm.io/
- asimuvPR 10y agoI had forgotten about them. Thanks for posting the link.
- vonnik 10y agoFwiw, we're testing a Dockerized the distro of DL4J. Runs on DCOS. https://imgur.com/a/CDTAc https://imgur.com/a/CDTAc https://imgur.com/a/6jlxi https://imgur.com/a/6jlxi We'll release in coming weeks.
- ymt123 10y agoHave you tried Sacred[1]? It definitely doesn't answer the "infrastructure for deep learning" challenge but it is helpful for understanding what experiments have been run/where did this model come from (including what version of the code/parameters produced it) [1] https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred
- cs702 10y agoOn a related note, I'm running a poll on deep learning frameworks: https://news.ycombinator.com/item?id=12391744 https://news.ycombinator.com/item?id=12391744
- josh_carterPDX 10y ago"Top performance thus requires top-of-the-line GPUs." Would be curious to see the data around the economics of the different options.
- komali2 10y agoEspecially metrics comparing cost of getting your own vs offloading to AWS or whatever. When is the "break even" point for buying your own?
- visarga 10y agoSo, a single GTX 1080 GPU deep learning box would come around 1500$, if you pay 0.7$/hr for your cloud server, you should buy if you use more than 1500/0.7 = 2142h. So, if you need more than 90 days of GPU time, you should probably buy your box. Of course, if the cloud server is slower than GTX 1080, then the benefit is multiplied. But ... your own box would not be scalable. You'd still need AWS to speed up training.
- josh_carterPDX 10y agoSeems to be a tough business to bootstrap. Interesting. Thanks for laying that out.
- coredog64 10y agoAFAIK, AWS is stuck several architecture revisions behind Pascal (most things I see say it's still a Kepler GK104). At a best guess, that 1080 is probably 2x faster than any single GPU AWS instance.
- freyr 10y ago> Like much of the deep learning community, we use Python 2.7 It's unfortunate that so much effort has been spent on bringing tools up to speed with Python 3, but some groups still insist on dragging their feet. I understand the motivation when we're talking about an established company with a huge legacy code base, but within the research community it's kind of embarrassing.
- daveguy 10y agoPython 2.7 is the present and future of scientific computing with Python. If there is one field in which a print statement is a critical feature it is quick interactive analysis prototyping. That won't change no matter how much Guido and co want it to change.
- nahumfarchi 10y agoCurious here, what difference does it make if it's a statement or a function?
- daveguy 10y ago4 extra keystrokes. If you could add a statement I would add "p" and change my muscle memory to use that. For me it doesn't even need the ">>" syntax, just quick display. Rapid prototyping needs rapid feedback. Edit: it's not just the quickness of a single statement. It's that the print statements are about 30-50% of the code when you are working this way.
- patcallier 10y agoTo be fair we've ingested a fair amount of python 2.7 research code for our Python 3 codebase and the print statements are the quickest of fixes. There are rarer actual gotchas, but 2to3 catches a fair number of them. We only switched for the machine learning project I'm working on right now as an experiment, but it's surprising how smoothly it's gone.
- mitbal 10y agoVery interesting article but I guess the scale is not for everyone. 1600 AWS GPU? I'll be lucky if my infra request for g2.8xlarge is approved.