6 ms·

As someone who missed the boat on this, is learning about this just for historical purposes now, or is there still relevance to future employment? I just imagin

As someone who missed the boat on this, is learning about this just for historical purposes now, or is there still relevance to future employment? I just imagine the OpenAI eats everyone's lunch in regards to anything AI related, am I way off base?

ksherlock 3y ago
Maybe last week's drama should have been a left-pad moment. For many things you can train your own NN and be just as good without being dependent on internet access, third parties, etc. Knowing how things work should give you insight into using them better.
- lamroger 3y ago
  I wonder if using APIs was more of a first to market move
- mnky9800n 3y ago
  Which drama of last week are you referring to? The one about the openai guy saying it's all just the data set? Or something else?
  - quchen 3y ago
    Their CEO was fired, hired by Microsoft, took a bunch of people with him, and is now back at the company
    - giardini 3y ago
      Did the "bunch of people" also return (from Microsoft to OpenAI)?
  - whatever_yeah 3y ago
    I must have missed the "dataset" news you're referring to, could you elucidate?
    - mnky9800n 3y ago
      i suppose its not news, but there was a long thread about it but it came from this post from june: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat...
- deleted 3y ago
  [deleted]
deepsquirrelnet 3y ago
This is about deep learning, of which LLMs are a subset. If you are interested in machine learning, then you should learn deep learning. It is incredibly useful for a lot of reasons. Unlike other areas of ML, the nature of deep learning is such that its parts are interoperable. You could use a transformer with a CNN if you wish. Also, deep learning enables you to do machine learning on any type of data, text, images, video, audio. Finally, it can naturally scale computationally. As someone pretty involved in the field, I lament that LLMs are turning people away from ML and deep learning, and following the misconceptions that there’s no reason to do it anymore. Large algorithms are expensive to run, have slow throughput and are still generally poorer performing than purpose built models. They’re not even that easy to use for a lot of tasks, in comparison to encoder networks. I’m biased, but I think it’s one of the most fun things to learn in computing. And if you have a good idea, you can still build state of the art things with a regular gpu at your house. You just have to find a niche that isn’t getting the attention that LLMs are ;)
- dafaqueue 3y ago
  I started off being really excited to learn, but as time went on I actually lost interest in the field. The whole thing is essentially curve fitting. The ML field is essentially an art more than a science and it's all about tricks and intuitions on different ways of getting that best fit curve. From this angle the whole field got way less interesting. The field has nothing deeper or more insightful to offer beyond this concept of curve fitting.
  - dbmikus 3y ago
    I've found this fun way to think of it: the goal is to invent a faster form of evolution for pattern recognition, learning, and autonomous task completion. I think one needs to consider it more like biology and a science than pure logic and math. We can discover things that work, and then after that we can study them to learn why they work, just like we don't fully understand the brain yet. I think there are some really cool problems, such as: 1. Is synthetic data viable for training? 2. How do you make deep learning agents that can do task planning and introspection in complex environments? 3. How do we efficiently build memory and data lookup into AI agents? And is this better/worse than making longer context windows?
GeneralMayhem 3y ago
The most important thing to learn for most practical purposes is what the thing can actually do. There's a lot of fuzzy thinking around ML - "throw AI at it and it'll magically get better!" Sources like Karpathy's recent video on what LLMs actually do are good anti-hype for the lay audience, but getting good practical working knowledge that's a level deeper is tough without working through it. You don't have to memorize all the math, but it's good to get a feel for the "interface" of the components. What is it that each model technique actually does - especially at inference time, where it needs to be well-integrated with the rest of the stack? In terms of continued relevance - "deep learning", meaning, dense neural nets trained to optimize a particular function, haven't fundamentally changed in practice in ~15 years (and much longer than that in theory), and are still way more important and broadly used than the OpenAI stuff for most purposes. Anything that involves numerical estimation (e.g., ad optimization, financial modeling) is not going to use LLMs, it's going to use a purpose-built model as part of a larger system. The interface of "put numbers in, get number[s] out" is more explainable, easier to integrate with the rest of your software stack, and more measurable. It has error bars that are understandable and occasionally even consistent. It has a controllable interface that won't suddenly decide to blurt corporate secrets or forget how to serialize JSON. And it has much, much lower latency and cost - any time you're trying to render a web page in under 100ms or run an optimization over millions of options, generative AI just isn't a practical option (and is unlikely to become one, IMO). I don't have a significant math or theoretical ML background, but I've spent most of the last 10 years working side by side with ML experts on infra, data pipelines, and monitoring. I'm not sure I could integrate the sigmoid off the top of my head, but that's not what's important - I've done it once, enough to have some idea how the function behaves, and I know how to reason about it as a black box component.
- mvkel 3y ago
  Awesome response, and a reasoned take.
- Breza 3y ago
  Terrific explanation, and it matches my experience running a data science team. I encourage my team to start with the simplest possible approach to every problem, which requires understanding how different algorithms work. Does this project require a t-test, XGBoost, a convolutional neural network, something else? What if we recode the dependent variable from numeric to binary?
ww520 3y ago
From an application perspective, it's more important to understand how the overall ML process work, the key concepts, and how things are fitted together. Deep Learning is a part of that. Lots of these are already wrapped in libraries and API, so it's a matter of preparing the correct data, calling the right API's, and utilizing the result.
lngnmn2 3y ago
[dead]
quickthrower2 3y ago
This would be like learning how your CPU / Memory works, even though JS is eating everyone's (web front-end) lunch. So yes if you are prompt engineering, and wondering why X works and sometimes it doesn't, and why any of this works at all, it is good to study a bit.
hedgehog 3y ago
Highly relevant if you want to work on ML systems. Despite how much OpenAI dominates the press there are actually many, many teams building useful and interesting things.
Slix 3y ago
I came here with the same question. After reading and learning these materials, will I have new job skills or AI knowledge that I can do something with?
two_in_one 3y ago
It's like calculus, nothing new in the last years, is it still important? The answer is still "Yes". After a glance, looks like too much for one book. Probably it was compressed with the assumption that reader already knows quite a lot. In other words it's not an easy reading.
niemandhier 3y ago
Someone will dominate the AI as a service marked, but there are so many applications for tiny edge ai that no single player can dominate all of them. OpenAI is for example not interested in developing small embedded neural networks that run on a sensor chip that real-time detects specific molecules in air.

ksherlock 3y ago

Maybe last week's drama should have been a left-pad moment. For many things you can train your own NN and be just as good without being dependent on internet access, third parties, etc. Knowing how things work should give you insight into using them better.

lamroger 3y ago
I wonder if using APIs was more of a first to market move
mnky9800n 3y ago
Which drama of last week are you referring to? The one about the openai guy saying it's all just the data set? Or something else?
- quchen 3y ago
  Their CEO was fired, hired by Microsoft, took a bunch of people with him, and is now back at the company
  - giardini 3y ago
    Did the "bunch of people" also return (from Microsoft to OpenAI)?
- whatever_yeah 3y ago
  I must have missed the "dataset" news you're referring to, could you elucidate?
  - mnky9800n 3y ago
    i suppose its not news, but there was a long thread about it but it came from this post from june: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat...
deleted 3y ago
[deleted]

deepsquirrelnet 3y ago

This is about deep learning, of which LLMs are a subset. If you are interested in machine learning, then you should learn deep learning. It is incredibly useful for a lot of reasons. Unlike other areas of ML, the nature of deep learning is such that its parts are interoperable. You could use a transformer with a CNN if you wish. Also, deep learning enables you to do machine learning on any type of data, text, images, video, audio. Finally, it can naturally scale computationally. As someone pretty involved in the field, I lament that LLMs are turning people away from ML and deep learning, and following the misconceptions that there’s no reason to do it anymore. Large algorithms are expensive to run, have slow throughput and are still generally poorer performing than purpose built models. They’re not even that easy to use for a lot of tasks, in comparison to encoder networks. I’m biased, but I think it’s one of the most fun things to learn in computing. And if you have a good idea, you can still build state of the art things with a regular gpu at your house. You just have to find a niche that isn’t getting the attention that LLMs are ;)

dafaqueue 3y ago
I started off being really excited to learn, but as time went on I actually lost interest in the field. The whole thing is essentially curve fitting. The ML field is essentially an art more than a science and it's all about tricks and intuitions on different ways of getting that best fit curve. From this angle the whole field got way less interesting. The field has nothing deeper or more insightful to offer beyond this concept of curve fitting.
- dbmikus 3y ago
  I've found this fun way to think of it: the goal is to invent a faster form of evolution for pattern recognition, learning, and autonomous task completion. I think one needs to consider it more like biology and a science than pure logic and math. We can discover things that work, and then after that we can study them to learn why they work, just like we don't fully understand the brain yet. I think there are some really cool problems, such as: 1. Is synthetic data viable for training? 2. How do you make deep learning agents that can do task planning and introspection in complex environments? 3. How do we efficiently build memory and data lookup into AI agents? And is this better/worse than making longer context windows?

GeneralMayhem 3y ago

The most important thing to learn for most practical purposes is what the thing can actually do. There's a lot of fuzzy thinking around ML - "throw AI at it and it'll magically get better!" Sources like Karpathy's recent video on what LLMs actually do are good anti-hype for the lay audience, but getting good practical working knowledge that's a level deeper is tough without working through it. You don't have to memorize all the math, but it's good to get a feel for the "interface" of the components. What is it that each model technique actually does - especially at inference time, where it needs to be well-integrated with the rest of the stack? In terms of continued relevance - "deep learning", meaning, dense neural nets trained to optimize a particular function, haven't fundamentally changed in practice in ~15 years (and much longer than that in theory), and are still way more important and broadly used than the OpenAI stuff for most purposes. Anything that involves numerical estimation (e.g., ad optimization, financial modeling) is not going to use LLMs, it's going to use a purpose-built model as part of a larger system. The interface of "put numbers in, get number[s] out" is more explainable, easier to integrate with the rest of your software stack, and more measurable. It has error bars that are understandable and occasionally even consistent. It has a controllable interface that won't suddenly decide to blurt corporate secrets or forget how to serialize JSON. And it has much, much lower latency and cost - any time you're trying to render a web page in under 100ms or run an optimization over millions of options, generative AI just isn't a practical option (and is unlikely to become one, IMO). I don't have a significant math or theoretical ML background, but I've spent most of the last 10 years working side by side with ML experts on infra, data pipelines, and monitoring. I'm not sure I could integrate the sigmoid off the top of my head, but that's not what's important - I've done it once, enough to have some idea how the function behaves, and I know how to reason about it as a black box component.

mvkel 3y ago
Awesome response, and a reasoned take.
Breza 3y ago
Terrific explanation, and it matches my experience running a data science team. I encourage my team to start with the simplest possible approach to every problem, which requires understanding how different algorithms work. Does this project require a t-test, XGBoost, a convolutional neural network, something else? What if we recode the dependent variable from numeric to binary?

ww520 3y ago

From an application perspective, it's more important to understand how the overall ML process work, the key concepts, and how things are fitted together. Deep Learning is a part of that. Lots of these are already wrapped in libraries and API, so it's a matter of preparing the correct data, calling the right API's, and utilizing the result.

lngnmn2 3y ago

[dead]

quickthrower2 3y ago

This would be like learning how your CPU / Memory works, even though JS is eating everyone's (web front-end) lunch. So yes if you are prompt engineering, and wondering why X works and sometimes it doesn't, and why any of this works at all, it is good to study a bit.

hedgehog 3y ago

Highly relevant if you want to work on ML systems. Despite how much OpenAI dominates the press there are actually many, many teams building useful and interesting things.

Slix 3y ago

I came here with the same question. After reading and learning these materials, will I have new job skills or AI knowledge that I can do something with?

two_in_one 3y ago

It's like calculus, nothing new in the last years, is it still important? The answer is still "Yes". After a glance, looks like too much for one book. Probably it was compressed with the assumption that reader already knows quite a lot. In other words it's not an easy reading.

niemandhier 3y ago

Someone will dominate the AI as a service marked, but there are so many applications for tiny edge ai that no single player can dominate all of them. OpenAI is for example not interested in developing small embedded neural networks that run on a sensor chip that real-time detects specific molecules in air.