4 ms·

As a software engineer, I disagree. Caveat, I haven't studied traditional ML and just went straight to DL. There is a lot of jargon and you do have to sit down

by carom 3y ago

As a software engineer, I disagree. Caveat, I haven't studied traditional ML and just went straight to DL. There is a lot of jargon and you do have to sit down and learn how things work, but once you do, deep learning is fairly simple. One thing that actually really bothers me is how much libraries (e.g. huggingface) are just config files masquerading as programming. It is just a class with 50 parameters and takes about 5 lines of code to make it do the thing, most of the time is spent figuring out what the parameters do. I think making lower level tools (fundamental building blocks) will make things more accessible to software engineers, as opposed to the high level wrappers being written. We can grab a library, read the docs, and put pieces together in an efficient way. We just need some core work horse libraries. Like if llama.cpp was a library at the same maturity as sqlite.

anon291 3y ago
The post is not talking about 'traditional ML' but rather only 'DL' (i.e. neural networks). One of the problems I see here is that it seems to me that the math education of many CS grads is woefully lacking. Indeed, deep learning math is basically senior-high-school level calculus. Backpropagation is a straightforwards application of the chain rule. There is really no surprising thing or deep insight. The math for deep learning has been set in place for more than fifty years at this point. The only change was the advent of large enough data and computer chips to process it. That being said, I don't think understanding how the `transformers` library works or composes is the same as understanding how 'deep learning' works. That's like saying knowing how to use GCC gives you a solid understanding of compilers. That's not a ding against those whose main experience with DL is using those libraries. I use many libraries doing things I don't fully understand. I'm not a graphics expert, but I use graphics programs every day, and graphics libraries regularly. That's fine. But using those libraries doesn't make you an expert in computer graphics.
- godelski 3y ago
  > Indeed, deep learning math is basically senior-high-school level calculus. Actually I find this claim problematic and incorrect. Yes, many parts of ML only require multivariate calculus[0] and those are the parts that most people are exposed to. BUT that doesn't mean there isn't a lot of math hiding around in the background. Even understanding something like activation functions take much higher level math as we got to get into topics such as topology, metric theory, and high dimensional statistics. You don't need this understanding to build models (as you also conclude) and most researchers don't understand much of this either tbh. But I think we need to be clear about these distinctions as model evaluation gets insanely complex. Complexity in evaluation only increases as our performance increases and a major problem we face today is we are marginalizing out important information. I'm sure that anyone that hacks around with LLMs or diffusion will understand how leaderboards are often noisy and how picking top models does not guarantee top performance (and why datasets get continually added). It's because evaluation requires more than boiling down performance to a single number. Here, math skills become extremely important. [0] btw, many will not find this even offered at their high school. In fact, this is often upper level math for many STEM undergraduates and even an elective in my Uni's undergrad CS program. So let's not be so belittling. Shame the education systems instead of people lacking opportunities.
- wenc 3y ago
  I don’t think I agree because it’s like saying accounting is just elementary level addition, subtraction and multiplication. But elementary school kids can’t do accounting. DL requires a certain degree of mathematical maturity to grasp.
thanhhaimai 3y ago
> There is a lot of jargon and you do have to sit down and learn how things work, but once you do, deep learning is fairly simple. This is my realization too. I think ML for SWE courses should focus on "translation" first. Like by "kernel" they mean this specific thing, not the normal meaning of kernel. This is similar to other fields like finance (which I'm working on). After you learn the language, it's actually not too terrible to understand.
- anon291 3y ago
  What's the normal meaning of 'kernel'? EDIT: Hacker news won't let me respond, but the answers below all seem to be because the original meaning has been lost on everyone. In English, the word 'kernel' means 'core'. An OS kernel is the core of an operating system. In linear algebra, the kernel of a matrix (or a linear transformation, same thing) is the set of vectors it maps to zero, which is also in a sense the 'core' of the mapping (in so far that zero can be seen to be at the 'core' of the vector space / number line). So actually, the definition is the same, it's just that the word kernel is rather rare, despite having a well understood meaning. Nevertheless, it is the kernel of many common English idioms such as a 'kernel of truth'. Going to ML kernels... the etymology is a bit convoluted. I believe they come from operator theory and support vector machines. But nevertheless, the 'normal' meaning of kernel works out because kernels are typically the core fundamental operations supported by a machine learning framework atop which the other operations are built. In that sense, regardless of the etymology, the name actually fits.
  - deleted 3y ago
    [deleted]
  - ozr 3y ago
    The bit of code at the center of an operating system that mediates access between userspace code and the hardware. Whereas in CUDA programming, a kernel is just the code running on the device. When I first heard kernel in relation to CUDA programming, I expected it to be a) the actual OS kernel, and then b) something akin to a GPU driver. I can see how NVIDIA got there, but it's not immediately obvious if you're coming from a SWE context.
jzombie 3y ago
> We just need some core work horse libraries. Do you mean like PyTorch, TensorFlow, or the others?
blackbear_ 3y ago
> we just need some core work horse libraries But these do exist: plain Jax or Pytorch only give you basic linear algebra, differentiation and some basic layers. And there's a plethora of more or less advanced libraries that add specific functionality, for example torch geometric for graph data and lightning to reduce boilerplate.
- beijbom 3y ago
  (I'm one of the Nyckel co-founders) Chiming in to say that this precisely our observation. The existing ML/DL libraries are not bad as far as those types of things go. In fact, Pytorch is an amazing library IMO. Especially compared to TensorFlow, Caffe and the stuff that came before that. But like George points out in the article, unlike "traditional" software, ML requires iteration, data management, monitoring, specific infra reqs, and so on. So our take was that libraries would never be enough, hence the SaaS offering.
  - mov 3y ago
    That's a super interesting article and point of view. So, in your opinion, what's missing?
catlover76 3y ago
[dead]

by carom 3y ago

anon291 3y ago

The post is not talking about 'traditional ML' but rather only 'DL' (i.e. neural networks). One of the problems I see here is that it seems to me that the math education of many CS grads is woefully lacking. Indeed, deep learning math is basically senior-high-school level calculus. Backpropagation is a straightforwards application of the chain rule. There is really no surprising thing or deep insight. The math for deep learning has been set in place for more than fifty years at this point. The only change was the advent of large enough data and computer chips to process it. That being said, I don't think understanding how the `transformers` library works or composes is the same as understanding how 'deep learning' works. That's like saying knowing how to use GCC gives you a solid understanding of compilers. That's not a ding against those whose main experience with DL is using those libraries. I use many libraries doing things I don't fully understand. I'm not a graphics expert, but I use graphics programs every day, and graphics libraries regularly. That's fine. But using those libraries doesn't make you an expert in computer graphics.

godelski 3y ago
> Indeed, deep learning math is basically senior-high-school level calculus. Actually I find this claim problematic and incorrect. Yes, many parts of ML only require multivariate calculus[0] and those are the parts that most people are exposed to. BUT that doesn't mean there isn't a lot of math hiding around in the background. Even understanding something like activation functions take much higher level math as we got to get into topics such as topology, metric theory, and high dimensional statistics. You don't need this understanding to build models (as you also conclude) and most researchers don't understand much of this either tbh. But I think we need to be clear about these distinctions as model evaluation gets insanely complex. Complexity in evaluation only increases as our performance increases and a major problem we face today is we are marginalizing out important information. I'm sure that anyone that hacks around with LLMs or diffusion will understand how leaderboards are often noisy and how picking top models does not guarantee top performance (and why datasets get continually added). It's because evaluation requires more than boiling down performance to a single number. Here, math skills become extremely important. [0] btw, many will not find this even offered at their high school. In fact, this is often upper level math for many STEM undergraduates and even an elective in my Uni's undergrad CS program. So let's not be so belittling. Shame the education systems instead of people lacking opportunities.
wenc 3y ago
I don’t think I agree because it’s like saying accounting is just elementary level addition, subtraction and multiplication. But elementary school kids can’t do accounting. DL requires a certain degree of mathematical maturity to grasp.

thanhhaimai 3y ago

> There is a lot of jargon and you do have to sit down and learn how things work, but once you do, deep learning is fairly simple. This is my realization too. I think ML for SWE courses should focus on "translation" first. Like by "kernel" they mean this specific thing, not the normal meaning of kernel. This is similar to other fields like finance (which I'm working on). After you learn the language, it's actually not too terrible to understand.

anon291 3y ago
What's the normal meaning of 'kernel'? EDIT: Hacker news won't let me respond, but the answers below all seem to be because the original meaning has been lost on everyone. In English, the word 'kernel' means 'core'. An OS kernel is the core of an operating system. In linear algebra, the kernel of a matrix (or a linear transformation, same thing) is the set of vectors it maps to zero, which is also in a sense the 'core' of the mapping (in so far that zero can be seen to be at the 'core' of the vector space / number line). So actually, the definition is the same, it's just that the word kernel is rather rare, despite having a well understood meaning. Nevertheless, it is the kernel of many common English idioms such as a 'kernel of truth'. Going to ML kernels... the etymology is a bit convoluted. I believe they come from operator theory and support vector machines. But nevertheless, the 'normal' meaning of kernel works out because kernels are typically the core fundamental operations supported by a machine learning framework atop which the other operations are built. In that sense, regardless of the etymology, the name actually fits.
- deleted 3y ago
  [deleted]
- ozr 3y ago
  The bit of code at the center of an operating system that mediates access between userspace code and the hardware. Whereas in CUDA programming, a kernel is just the code running on the device. When I first heard kernel in relation to CUDA programming, I expected it to be a) the actual OS kernel, and then b) something akin to a GPU driver. I can see how NVIDIA got there, but it's not immediately obvious if you're coming from a SWE context.

jzombie 3y ago

> We just need some core work horse libraries. Do you mean like PyTorch, TensorFlow, or the others?

blackbear_ 3y ago

> we just need some core work horse libraries But these do exist: plain Jax or Pytorch only give you basic linear algebra, differentiation and some basic layers. And there's a plethora of more or less advanced libraries that add specific functionality, for example torch geometric for graph data and lightning to reduce boilerplate.

beijbom 3y ago
(I'm one of the Nyckel co-founders) Chiming in to say that this precisely our observation. The existing ML/DL libraries are not bad as far as those types of things go. In fact, Pytorch is an amazing library IMO. Especially compared to TensorFlow, Caffe and the stuff that came before that. But like George points out in the article, unlike "traditional" software, ML requires iteration, data management, monitoring, specific infra reqs, and so on. So our take was that libraries would never be enough, hence the SaaS offering.
- mov 3y ago
  That's a super interesting article and point of view. So, in your opinion, what's missing?

catlover76 3y ago

[dead]