6 ms·

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months t

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Seems like universally these posts have terrible curse of knowledge [1]. To be fair, I know only a light-to-moderate amount about machine learning and very little about differential equations, but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. Maybe I just need to...try harder? I wonder how many people would be excited but are totally lost by these posts. To that point, if anybody has a recommendation for a gentle introduction to these topics (preferably Julia), I'd be most appreciative. [1] https://en.wikipedia.org/wiki/Curse_of_knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge

jimsimmons 5y ago
I consider myself to be reasonably good at ML and work for one of the cool labs and this stuff is impenetrable to me. Stuff like this are deceptively written and a cynic might say that they are primarily to show off rather than help.
agumonkey 5y ago
What's funny is how most of them are revolving around a similar culture of concepts (ODE, automated differentiation, analysis) which is very unlike mainstream computing. ps: I need to read those adjoints articles.
uoaei 5y ago
> but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly understood (by those who read the papers, where I think it is an appropriate assumption that people who want to understand this stuff would do) to be effectively infinite-depth neural networks where every layer represents the same function.
- diskzero 5y ago
  Have you been in interview loops or worked with bread and butter data scientists performing common tasks? I am curious what your view of what most data scientists do day in and day out?
  - uoaei 5y ago
    What about tasks being common makes a light-to-moderate understanding of machine learning sufficient? Processes initiated by data scientists during the execution of their role will tend to fail silently. What is meant here is, throwing an inappropriate model at otherwise good data produces unreliable (catastrophic in certain situations) results, but produces results nonetheless. Without the proper discernment of the reliability of the results, we have an unequivocal failure to execute the role. This is the oft-unmentioned companion to, but decidedly more insidious than, the "garbage in, garbage out" (i.e., right model, wrong data) aphorism. It is up to the person performing this operation to deduce whether or not the conclusions are trustworthy. I don't see how someone can be confident of this without either relying on a pre-defined workflow verified by someone else qualified to assess the consequences, or to have those qualifications themselves. What follows is a contrived example, but illustrative of the problem: Consider e.g. user privacy: it is by now well-known that e.g. embedding vectors (or even merely the relationships between them) can leak a lot of information about the person or object it represents. It is not enough to understand how the forward pass of such a model commences, but also what is stored in those representations, which, having gone through a master's with quite a few people who now call themselves data scientists, I am not confident is commonly understood.
    - diskzero 5y ago
      This is helpful, and I agree with you. There are some "data scientists" who are able to use PyTorch, TensorFlow, etc. and modify code in a Jupyter Notebook without knowing the larger ramifications of the work they are doing.
- eigenspace 5y ago
  This is a pretty shitty, non-constructive response. I think neural differential equations are not easy to wrap one's head around even if you have a solid understand of deep learning and differential equations. Sure, if they spent a lot of time wading through the literature they'd probably understand fine, but the point they were making was that the post was quite unapproachable without having delved into the specific literature on neural differential equations. I think this is a reasonably valid complaint, and does not warrant you implying that they don't deserve their job.
ViralBShah 5y ago
The SciML tutorials are a good start: https://github.com/SciML/SciMLTutorials.jl https://github.com/SciML/SciMLTutorials.jl And also the 18.337 lecture notes (probably also has videos available): https://github.com/mitmath/18337 https://github.com/mitmath/18337

jimsimmons 5y ago

I consider myself to be reasonably good at ML and work for one of the cool labs and this stuff is impenetrable to me. Stuff like this are deceptively written and a cynic might say that they are primarily to show off rather than help.

agumonkey 5y ago

What's funny is how most of them are revolving around a similar culture of concepts (ODE, automated differentiation, analysis) which is very unlike mainstream computing. ps: I need to read those adjoints articles.

uoaei 5y ago

> but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly understood (by those who read the papers, where I think it is an appropriate assumption that people who want to understand this stuff would do) to be effectively infinite-depth neural networks where every layer represents the same function.

diskzero 5y ago
Have you been in interview loops or worked with bread and butter data scientists performing common tasks? I am curious what your view of what most data scientists do day in and day out?
- uoaei 5y ago
  What about tasks being common makes a light-to-moderate understanding of machine learning sufficient? Processes initiated by data scientists during the execution of their role will tend to fail silently. What is meant here is, throwing an inappropriate model at otherwise good data produces unreliable (catastrophic in certain situations) results, but produces results nonetheless. Without the proper discernment of the reliability of the results, we have an unequivocal failure to execute the role. This is the oft-unmentioned companion to, but decidedly more insidious than, the "garbage in, garbage out" (i.e., right model, wrong data) aphorism. It is up to the person performing this operation to deduce whether or not the conclusions are trustworthy. I don't see how someone can be confident of this without either relying on a pre-defined workflow verified by someone else qualified to assess the consequences, or to have those qualifications themselves. What follows is a contrived example, but illustrative of the problem: Consider e.g. user privacy: it is by now well-known that e.g. embedding vectors (or even merely the relationships between them) can leak a lot of information about the person or object it represents. It is not enough to understand how the forward pass of such a model commences, but also what is stored in those representations, which, having gone through a master's with quite a few people who now call themselves data scientists, I am not confident is commonly understood.
  - diskzero 5y ago
    This is helpful, and I agree with you. There are some "data scientists" who are able to use PyTorch, TensorFlow, etc. and modify code in a Jupyter Notebook without knowing the larger ramifications of the work they are doing.
eigenspace 5y ago
This is a pretty shitty, non-constructive response. I think neural differential equations are not easy to wrap one's head around even if you have a solid understand of deep learning and differential equations. Sure, if they spent a lot of time wading through the literature they'd probably understand fine, but the point they were making was that the post was quite unapproachable without having delved into the specific literature on neural differential equations. I think this is a reasonably valid complaint, and does not warrant you implying that they don't deserve their job.

ViralBShah 5y ago

The SciML tutorials are a good start: https://github.com/SciML/SciMLTutorials.jl https://github.com/SciML/SciMLTutorials.jl And also the 18.337 lecture notes (probably also has videos available): https://github.com/mitmath/18337 https://github.com/mitmath/18337