9 ms·
Cutting Edge Deep Learning for Coders, Part 2
- jph00 8y agoJeremy from fast.ai here. I've posted a summary of the course materials for anyone who's interested : http://www.fast.ai/2018/05/07/part2-launch/ http://www.fast.ai/2018/05/07/part2-launch/ Let me know if you have any questions about the material or approach. There's also a discussion for the course here: http://forums.fast.ai/c/part2-v2 http://forums.fast.ai/c/part2-v2
- abcdcba 8y agoHi Jeremy - For someone who doesn't like to watch lectures, will you guys ever be releasing a textbook?
- jph00 8y agoI might be helping author a textbook based on our upcoming introduction to machine learning MOOC. Which maybe one day would lead to a deep learning book - but that would be years away! (BTW, I used to think I didn't like videos for learning, but actually I now think some material works best in this format. E.g. in this case we're walking through interactive notebooks, and you can follow along too. Some of the material is animated, and often I'm building up drawings as I talk. Maybe it won't be for you, but it might be worth giving it a go to see...)
- paultopia 8y agoJust another thought on this, as someone slowly trying to work his way through part 1 (and generally very happy with it, thanks for this amazing resource!): videos are great when they're really short. It's tough for working people to find enough unbroken time to watch an hour or two long video, like the ones in fast.ai (as amazing as they are), especially with frequent pauses for note-taking and such. But it's much easier to handle smaller chunks. This is probably my only critique of/gripe about fast.ai---the videos pack a lot of topics in, and they could be broken at topic boundaries to make them easier to use.
- jph00 8y agoThanks for the feedback! One thing you may be interested in, if you haven't seen it, is that you can right-click the video and copy a link to the current time code, so you can always go back to where you were (or just pause the video in a tab and return later). Personally, I really hate the Coursera approach of lots of separate short videos - I totally get that some people like it, and that's fine, but it's not something I want to do myself.
- paultopia 8y agoOh, I did not know you could link to specific time slices! That's a big help, thanks!
- Dibes 8y agoanother thing I used a ton when going through part 1 was being signed into my google account, google kept track of my progress through the video and if I exited out and came back on a different machine it would resume where I last left off, then I would back up a few minutes to build that context back up (or attempt to anyways).
- dominotw 8y agoI have the same problem. Too traumatized (when I was young) by a person teaching in a lecture format. I like self discovery and tinkering.
- fantispug 8y agoI'm traditionally a textbook person too, but I really like fast.ai. You get to see how to tinker with things in real time (which is often clunky in a textbook), and there's a lot of informal folklore that people don't normally write in academic textbooks. If you're dead set against video you could just work through the notebooks yourself and read the referenced papers on the website.
- jonahx 8y agoHey Jeremy, thanks for making these courses! I was curious if you could offer any points of comparison between the fast.ai courses and the Udacity ML nanodegree course (or any other MOOCs you have opinions about).
- jph00 8y agoI haven't taken the course - and I'm hopelessly biased anyway so you shouldn't trust my comments. However a lot of our students have done it. Here's a couple of their comments (search the linked thread if you want more): - http://forums.fast.ai/t/how-has-your-journey-been-so-far-learners/6480/33 http://forums.fast.ai/t/how-has-your-journey-been-so-far-lea... - http://forums.fast.ai/t/how-has-your-journey-been-so-far-learners/6480/45 http://forums.fast.ai/t/how-has-your-journey-been-so-far-lea... Some folks have taken the Udacity flying car and self-driving car courses as well as fast.ai, and had success building on that combination, e.g. http://forums.fast.ai/t/meet-greet-thread-introduce-yourself/13345 http://forums.fast.ai/t/meet-greet-thread-introduce-yourself...
- syncopate 8y agoI am not Jeremy but I have taken part in the current course and also took some lessons by Siraj (who also does lessons at udacity), Andrew NG, and Karpathy. I think all of them are awesome teachers. Like Siraj's lessons, the fastai course is full of practical examples, whereas Ng/Karpathy's courses are more theoretical. What makes the fast.ai course unique compared to Siraj (who covers a lot of topics very briefly) is that Jeremy delves into depth, taking a long time explaining things, so you have a lot of opportunities to understand what's going on. In the 2018 course there is actually not that much different content compared to the 2017 course, rather Jeremy took the time and tried to make everything easier for everyone to understand and built a better python library. Also another unique point about Jeremy is that he and Rachel want not only to teach people but rather inspire a community of people working together in teams. The fast.ai library will not be just a project for the course, it will be something that a lot of people will use and contribute to to make pytorch better.
- bitL 8y ago
- kleolon 8y agoHi Jeremy. Just wanted to ask a quick question if you recommend any prerequisites before taking this course? I am a second year Computer Science student looking to get into AI, and I was just wondering if this would be a good starting point for me, or should I take some other courses and learn something else before? Thank you.
- jph00 8y agoThis linked course is part 2, so this isn't a good starting point. But part 1 is! :) Here's the link: http://course.fast.ai http://course.fast.ai . We assume you're familiar with python and somewhat familiar with numpy. If you're not sure if you're ready, you can always just start the first lesson and see how it goes. There are plenty of resources on the forum to help fill in specific skill gaps as they come up.
- ezhil 8y agoJeremy and Rachel are very active on Twitter. I learn a lot from their tweets too! Thanks!!
- RSchaeffer 8y agoHey! Thanks for these great courses and materials! How much additional math (beyond high school and introductory college courses) do these courses teach? For example, if I were to take both courses, would I be able to understand the papers published by Surya Ganguli (e.g., The Emergence of Spectral Universality in Deep Networks, Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net)?
- poppingtonic 8y agoOne year of Python, with high school and introductory college courses. Some basic linear algebra, familiarity with logarithms etc.
- RSchaeffer 8y agoI think you misread my question. I'm asking what math students will learn, not what math students should already know.
- bitL 8y agoGram matrix, some non-linear optimization. Deep Learning is simple, complexity comes from loss function, trying to adjust weights of under-represented categories, different LEGO blocks in building your network and seeing if the particular non-linear optimization works in your case or not. You can go super deep with state-of-art math research in reading about "why do we think Deep Learning works, when it shouldn't", which is mentioned by Jeremy.
- jph00 8y agoGanguli's papers are at the more "math-y" end of the spectrum when it comes to DL papers. So don't worry if you're finding them a bit unapproachable. In part 2 of the course I provide quite a bit of advice about how to approach papers in general, and you'll get plenty of practice in reading and implementing papers - but we don't cover the specific math in this particular papers. My view is the best approach to the math in papers is to generally learn what you need as you get there. It's nearly impossible to know all the math that covers every paper you'll come across, but if you learn the meta-skill of how to learn it on demand, then you'll be just fine! :)
- gumballhead 8y agoThank you for creating these courses and putting them online! I started part 1 a few weeks ago. I couldn't agree more with your teaching style. I'm someone who started college as a computer science major and couldn't finish, because it felt like I was pounding my head on the wall trying to learn abstract math when what I really wanted to do was build stuff. I finished my degree with economics, went to work in finance, and worked my way to being a developer by first doing advanced Excel, then Access, then database servers, then web development, then native development, and now more math heavy research and development. As the need arises, I keep going further down the stack and gaining more understanding. It's much more natural (for me) to learn that way.
- geezerjay 8y ago> because it felt like I was pounding my head on the wall trying to learn abstract math when what I really wanted to do was build stuff. But you do need to know abstract math to build stuff, particularly if what you want to build is deep learning stuff, both models and implementations. For example, how do you expect to understand how to minimize an utility function if you have no idea of what a gradient is, how you calculate it, and why you want to descend through it.
- jph00 8y ago> For example, how do you expect to understand how to minimize an utility function if you have no idea of what a gradient is, how you calculate it, and why you want to descend through it. The course teaches all those things - as the comment you're replying to states, you go deeper and deeper during the course to understand all the details. There's been a lot of research into teaching strategies that shows that this is often a more effective approach for many people than the bottom up approach widely used in math and CS. It doesn't mean that you learn any less of the foundations - just that it's in a different order.
- geezerjay 8y ago> There's been a lot of research into teaching strategies that shows that this is often a more effective approach for many people than the bottom up approach widely used in math and CS. I seriously doubt that anyone can effectively learn linear algebra, multivariate calculus, optimization and regression models from an onlone tutorial on deep learning. These are subjects whose basics alone take multiple semester-long courses. If a bottom-down approach was remotely effective, no one would bother teaching the basics.
- zengid 8y agoThanks Jeremy (and Rachel) for everything y'all have done; I learned a lot just from watching part 1! Somewhat tangential question, but how do you feel about sticking with PyTorch for future lessons/research work? What are your thoughts on the new Swift integration with TensorFlow?
- jph00 8y agoWe're so early in the software and tools cycle for DL that it's unlikely anything that exists today will remain relevant in its current form for very long. You can see that every couple of years the entire DL ecosystem is getting replaced - I don't think that will change. I'm glad you asked about Swift. It's one of the two directions I'm most excited about for the next stage of DL software (the other being javascript!) Swift is a much better language than Python, and having autograd and tensor types built in will be quite transformative. But it has nothing like the same data science library and tooling, so it'll take time to be a real option for most people. Javascript has the potential to make DL accessible even for those without a discrete GPU, and allows people to get started even without installing a tool-chain. It also allows more work to be pushed to the client - possibly even on mobile devices.
- ScottBurson 8y agoHere's a vote for Julia over Swift.
- source99 8y agoI can't say enough good things about the fast.ai courses. They may not be for everyone but they are definitely for me. The teaching style is a perfect match for my learning style. The style is to get some code working and then dive deeper and deeper to understand it better and better. Note this link is for part 2 of the course which is significantly more advanced than part 1. Start with part 1 if if you are new.
- ironfootnz 8y agoThat's a really good upgrade.
- doall 8y agoI watched all of the 2018 Part 1 videos and it was not my style. One thing I liked is the positive attitudes of Jeremy and Rachel, but there was not much theory and a lot of time was spent on questions and answers for the participants of the lecture that I don't think necessary. I wanted to see definitions and "why", but the course spends too much time on "how". Sometimes I see "why" in the lecture, but many times it is just an answer for the question from a random participant and the answers seemed not well prepared, so it makes it hard to deeply understand the content. I think fast.ai (at least Part 1) may be good after taking other deep learning courses and before participating in the Kaggle competitions that when you are already familiar with the theories.
- yumraj 8y agoI'll have to agree with this one. I had watched, and done most of the assignments, in the 2017 part-I of the course. While I did get stuff working and got results, I was left unsatisfied. I recently went through the videos, of Andrew Ng's Coursera course to get feel for the theory and intuition,and I'll be taking another stab at fast.ai courses to compare his assignments with Andrew Ng's.
- minimaxir 8y agoGranted, each person has their own learning style. Some do better with theory, some do better with practical application. In my case, I stopped doing data MOOCs entirely once I realized I learned 10x faster just by reading documentation and working on a project from the bottom-up, since MOOCs rarely highlight the annoying gotchas present in real-world data applications.
- jph00 8y agoTop-down vs bottom-up depends to some extent on what you enjoy, vs what you need to be patient about. With our top-down approach you do get to all of the 'why', but only after understanding 'how'. So it's good for people who want to get started doing stuff right away, and don't mind waiting a bit to understand the details. It means you can start experimenting and building a good intuition for training models, which I believe is the most important skill for a practitioner. On the other hand, with the bottom-up approach of Andrew Ng in deeplearning.ai, you start with a lot of 'why', and later on get to 'how' (although in less detail and fewer best practices than we show). So it's good for people who want to understand the theory right away, and don't mind waiting a bit to understand how to use it. A lot of our students did Andrew's course after ours, and many did it in the reverse order. All have reported finding the combination more helpful than either on their own. When we describe 'why' it's mainly with code, whereas with Andrew it's mainly with math - so which you prefer will also depend on which notation and framework you're more comfortable with. (But I promise - you do get all the 'how' with us, particularly in part 2! Our students have gone on to OpenAI, Google Brain, and senior AI leadership positions at well known startups, as well as writing and implementing new papers. Here's an example of a student who just implemented a paper that was released within the last month: https://sgugger.github.io/deep-painterly-harmonization.html#deep-painterly-harmonization https://sgugger.github.io/deep-painterly-harmonization.html#... )
- apohn 8y agoJeremy - First of all, I just want to say thanks to you and Rachel for the course(s). I worked through part 1 (2017 version) last year and it was fantastic. I've got part 2 on my to-do list for this year. Looking at the 2018 part 1 course, quite a lot has changed (from Keras to Pytorch?) from a code perspective in comparison to the 2017 part 1 course. If I want to start part 2, does it make sense to go-ahead with the 2018 version? Are there any 2018 part 1 lessons you recommend to bridge any gaps from the 2017 part 1 before I start 2018 part 2? Thanks!
- gcmac 8y agoNot Jeremy (obviously) but having taken both versions of the course I would definitely recommend jumping into the 2018 version. The core content is the same (image recognition, recommender systems, NLP) but the 2018 version uses more cutting edge tools and has a tighter feel (as a v2 of anything usually does).
- imnotadoctor999 8y agoThe field is changing so quickly, but fast.ai is always ahead of the curve teaching modern techniques! Hats off to them for being able to keep up, AND share that with others.
- apohn 8y agoThanks. I'll go straight to 2018 part 2. Looking forward to it!
- 40four 8y agoWow great share! I was unaware of these courses! Great work @jph00
- sampathweb 8y agoI have taken the classes and also did Udacity Nano Degree and Deeplearning.ai. All of them help you in your journey of mastering DL. But Fast.AI part-2 really teaches you to cutting edge of DL through a number of practical examples. Don't wait and join other fellow learners at forums.fast.ai
- YeGoblynQueenne 8y ago>> Welcome to the new 2018 edition of fast.ai's second 7 week course, Cutting Edge Deep Learning For Coders, Part 2, where you'll learn the latest developments in deep learning, how to read and implement new academic papers, and how to solve challenging end-to-end problems such as natural language translation. I would really like to know how to solve natural language translation. I think everyone would. Many people have been trying to solve this devilishly hard problem for several decades and failed. So I'm really curious how fast.ai has finally manged to do it.
- stochastic_monk 8y agoMachine translation is not solved, but it's reached some surprisingly improved benchmarks for accuracy, so while it's a little presumptuous to call it solved, it's not the most egregious exaggeration I've heard about machine learning this week.
- YeGoblynQueenne 8y agoEveryone's doing it, so it's OK?
- pixelHD 8y agoWell, I looked at the summary, and they're implementing a Seq2Seq model for this. It is what I think of as an archetype for machine translation and chat bot tasks. Quite a few new network architectures in this space have been updates to this model, which uses an RNN encoder and a decoder, along with attention between them and a beam search for better results. I wouldn't call this model a solution for natural language translation, nor would anyone else. But I think fast.ai meant that they're going to explain and go through this model, and how it's helped bring a new generation of models with good performance in this particular space.
- jph00 8y agoYup it's multi-layer bidir seq2seq with attention, and a few tricks like teacher forcing. Same as Google Translate. Their version takes a long time to train on a lot of GPUs, so we simplify it by using less layers, and a smaller, simplified corpus (it only contains questions, and limits them to 30 words long). By "solve end-to-end problems" I only mean that we show how to do the whole process from beginning to end - I didn't mean to imply that the final model would be human-equivalent or perfect or anything like that.
- DrNuke 8y agoHi Jeremy, Thanks again to you and Dr Rachel for these amazing courses and the support to the community. My question is the following: are you thinking of an organic fast.ai connected hub or foundry acting as a distillator, that is disseminating & implementing in / with / through the fast.ai library the most interesting research papers coming out day in day out from arxiv.org and events? It is nearly impossible to follow the progress without clear state-of-the-art perspective these days, which you have in abundance indeed.
- jph00 8y agoWe're certainly looking for ways to help develop the community. Obviously http://forums.fast.ai http://forums.fast.ai is a good starting point, and is a very active and supportive community nowadays. But I'm interested in finding ways to support face-to-face communities too. Some students are already doing that - for instance AI Saturdays (https://nurture.ai/ai-saturdays https://nurture.ai/ai-saturdays) has study groups in 50 cities now, and I'm told ~5000 people have been involved in them. I'm always open for ideas in other ways we can help people get involved and stay up to date. And I'm very interested in supporting students who are trying to build these things in their local community.
- tedsanders 8y agoQuestion: What is the job market like for folks who spend two months learning deep learning, either through a class like fast.ai or a bootcamp like Insight AI? What sorts of companies hire these people and how much do they pay? I know it varies, but I'm just trying to get a sense to calibrate my expectations of the current market. Thanks for any perspective anyone can offer.
- DrNuke 8y agoAs an outsider and / or away from the SV, where such an environment thrives and opportuinities are a few dozens, your mileage may vary a lot. Personal network is always better to refer to than shots in the dark though. In general, you might need showcasing your knowledge through marketable case studies in a sub-domain you are interested in, so that interviews will come from a shared context.
- jph00 8y agoExactly right - it's all about your portfolio. We spend a lot of time throughout the course talking about ways to build that portfolio, and folks on the forums support each other with feedback and help on these projects. Completing a MOOC isn't going to help you land an interview for a job, generally speaking (although if it's any good, it should certainly help you once you do get an interview!) However if you use your study time to, for example, create useful projects that you make available on github, web apps that you host on heroku, deep technical blog posts you post on medium, etc then you should definitely get plenty of interview opportunities. I've seen many many students go through this process, so I know it works. That includes students that hadn't been getting interviews or job offers - until I convinced them to spend time building a portfolio, and then they got multiple offers from top companies. (There are some companies - not too many thankfully - that strictly require a PhD for DL applicants. I've noticed that such companies, that focus on credentials over actual work output, seem to overlap a lot with companies that turn out to have toxic cultures. So I'm not sure you should worry too much about them...)
- nonytoms 8y agoHire a professional hacker that can help you with all your hacking problems.Clear criminal records, university upgrade, Facebook hack, instagram hack, WhatsApp hack, bank transfer,credit card top etc Contact us via geraldhacks30@gmail.com or ICQ: 703132619 or WhatsApp +14696660879
- kriro 8y agoI have nothing to say other than that I loved DL1 and I'll dive into DL2 right away. I really like the overall philosophy ("see DL is approachable and doable, just do it") and I love that they encourage you to read papers even if they seem hard and to translate math to code etc. etc. Fantastic job fast.ai team. It's pretty damn amazing that they build a library (and also the new fastai.text) that goes one level of abstraction beyond PyTorch with the goal of implementing interesting/helpful papers ASAP and because they wanted to make their teaching even more efficient.