5 ms·
Jeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this
by jph00 9y ago
Jeremy here. Here to answer any questions or comments that you have.
But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish https://github.com/parrt/bookish . (We tried Katex, Mathjax, and pretty much everything else but nothing rendered everything properly).
I've never found anything that introduces the necessary matrix calculus for deep learning clearly, correctly, and accessibly - so I'm happy that this now exists.
- jules 9y agoWhat do you think about the index notation physicists use for tensor calculus?
- jph00 9y agoI don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me. The only tensor notation I've been happy with is that used in J (http://www.jsoftware.com http://www.jsoftware.com), which is simple, flexible, and concise. There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS+BOOK/Kolda-Bader-SAND2007-6702.pdf http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...
- deleted 9y ago[deleted]
- jules 9y agoThe notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply: A_ij = B_ik C_kj Differentiating this with respect to variable l: ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj) By writing out indices you can just use the rules for scalar derivatives.
- jph00 9y agoOh in that case I'm just confused - that's what I know as Einstein notation. Modern physics papers seemed to use much more complex notation, but I probably just misunderstood. If we're talking about Einstein notation, then I'm a fan - `np.einsum()` is often a great way to create fast tensor computations with minimal code.
- mlevental 9y agoindex notation isn't useful for calculation it's useful for algebra i.e. if you want closed form solution to tensor equations. my distinct impression is that in ML no one cares about that because eventually everything gets a numerical treatment.
- Mangalor 9y agoThis is really great, thanks.
- parrt 9y agoTerence here. Jeremy's role was critical in terms of direction and content for the article. Who better than he to describe the math needs for deep learning. :)
- MarkMMullin 9y agoGlad to see you in this space Terrance! Been a long time since the traveling parser revival and beer tasting festival!
- jacobolus 9y agoTypographic advice: the body text has very long lines in a desktop browser, which makes it a bit slow and tiring to read. I’d say the ideal is somewhere between 1/2 and 2/3 this length. I’d recommend keeping the same width on screen but bumping the font size up by 30%. As an extra minor nit, italicizing functions like sin, etc. is also somewhat unconventional in mathematical typesetting.
- jph00 9y agoThanks for the advice!
- parrt 9y agoI agree that the font should be bigger. I need to learn more CSS in order to switch between font sizes per platform. The font of the text is easy but all of the images were generated from latex using a specific font size. I need to scale the in-line equation images as the font size bumps up.
- dbetteridge 9y agothe magic incantation here is probably media queries @media (max-width: 768px) { p { font-size: 1rem; } }
- parrt 9y agoWe have to also adjust the image sizes for the in-line equations. That’s what I need to figure out :)
- posterboy 9y agoit's a variable name for a polymorphic function, taylor series, euler formular, etc. depending on context. This is a significant difference to singular types, but being a variable name to an abstract concept is in principle no different to typesetting x. This goes neatly with "everything is an objectreference" and might be more of a programmer's perspective.
- jtmcmc 9y agoThis is great! my graduate advisor a long time ago made a really great matrix calculus study sheet for me that was absolutely invaluable in learning ML ( i mean really this is great for not just DL but all sorts of reasoning in ML)
- smith-kyle 9y agoHi Jeremy, thanks for creating so many great learning resources. Myself and a fellow ML enthusiast are starting a deep learning meetup here in Phoenix next month. Do you have any advice for creating a welcoming environment for people to learn/teach? Thanks!
- jph00 9y agoProbably better asking that at http://forums.fast.ai http://forums.fast.ai , since a lot of participants there have set up local meetups, and some of them have gotten pretty big. I haven't created a meetup myself - my focus has been on the online course and community, frankly.
- edhu2017 9y agoIn the article, you say the gradient of 2x + y^8 is [1, 7y^8]. Shouldn't it be [2, 7y^8]?
- archgoon 9y agoQuestion; when I learned Vector Calculus back in college, I used Marsden & Tromba as our text book where they equate the derivative of a function from R^n->R^m with the Jacobian. Is matrix calculus the same thing, just a slightly different notation?
- TheArcane 9y agoThere are discrepancies between the HTML and pdf version. Eg. In the first line in 'References' HTML: "When looking for resources on the web, search for “matrix calculus” not “vector calculus.” " Latex/pdf: "When looking for resources on the web, search for “elements” not “elements” "
- parrt 9y agoWhoops. thanks. translator error. I'll fix it.
- 2paisay 9y agoERRATA: In the start of Matrix Calculus section, shouldn't the first value in gradient of "g" be 2 instead of 1 i.e., [2,8y^7]