3 ms·

Follow the HuggingFace Colab notebooks. They are well-written and language-related AIs are a great way to get started because you'll naturally have a feeling fo

Follow the HuggingFace Colab notebooks. They are well-written and language-related AIs are a great way to get started because you'll naturally have a feeling for what it should produce. Afterwards, do a statistics class. Most algorithms these days are based on softmax, meaning the cross-entropy between two discrete/continuous probability distributions. There's a lot of choice in which distribution to use to model what and it will have strong effects on your gradients and, hence, training trajectory. Concepts like shannon information and entropy are also very helpful for you to monitor training progress. Typical loss values will do exponential annealing and it'll be difficult to see further progress. But if you still reduce the bits of entropy in your classifier, learning is still going well. So you need to understand what to visualize and how to calculate that. As for implementing research publications, maybe start with easy mode and go to paperswithcode.com . There, you will find papers AND their source code, so that you can look at how others implemented their paper. As for FastAI and Kaggle, my personal impression is that it's mostly for toy problems. No real AI researcher would be willing to disclose their full source code to an international megacorp like H&M for a measly $15k in price money, yet similar terms appear to be the default on Kaggle: https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/overview/prizes https://www.kaggle.com/competitions/h-and-m-personalized-fas... https://www.kaggle.com/competitions/dfl-bundesliga-data-shootout/overview/prizes https://www.kaggle.com/competitions/dfl-bundesliga-data-shoo... https://www.kaggle.com/competitions/feedback-prize-effectiveness/overview/prizes https://www.kaggle.com/competitions/feedback-prize-effective... EDIT: Also, I strongly disagree with course.fast.ai on these points: "Myth (don’t need): Lots of math, Lots of data, Lots of expensive computers" To train a state of the art ASR AI, you need roughly 100x A100 for a month, 100,000+ hours of audio recordings, and math knowledge to find a maximum likelihood path through a logit matrix. Unless, of course, you're only working on toy problems.

inciampati 4y ago
Great suggestions and perspective! Thanks!
shoo 4y ago
> to train a state of the art ASR AI, you need roughly 100x A100 for a month, 100,000+ hours of audio recordings, and math knowledge to find a maximum likelihood path through a logit matrix. Do you think there is a distinction between the kinds of problems that take some kind of "raw" signal data (audio, images etc) as input, where deep learning approaches appear to be fruitfully applied, and other kinds of problems that appear in many places in business and the public sector where the input space is not some kind of raw signal data but instead tabular data. I have heard some people argue that the latter kinds of tabular-data style problems can be effectively tackled with a variety of statistical methods, and that deep learning style approaches do not offer an advantage. Maybe fast.ai is commenting on the latter class of problem.
- fxtentacle 4y ago
  I agree with you :) Most business-style "automate my Excel" problems can be solved pretty well with regular statistical models, meaning you don't need AI to solve them. So my impression is that fast.ai teaches you how to use AI methods, but with example problems that didn't need AI to be solved well.
robga 4y ago
fast.ai openly declares it doesn't have complete pipelines for audio, especially ASR. They're also very open that areas like ASR and RL take a ton of compute to replicate SOTA. This shouldn't be used to substantiate any sort of criticism of fast.ai the library since it is just wrappers on pytorch. Maybe on the course.fast.ai page they could make some sort of disclaimer that some areas like ASR and yet to achieve the quantum leap of performance that we've seen in other domains. I disagree completely that fast.ai is for toy problems. It's a development tool that in my experience provides for more rapid iteration than starting in vanilla pytorch. Use it for toys, for challenges, for research, or in industry. You can always rewrite your solution in pytorch once stable if there is benefit in doing so. I agree with others that fast.ai is a great place to start.
whimsicalism 4y ago
ASR is one of the hardest tasks for a novice to ML/DL to do.
- fxtentacle 4y ago
  In my opinion, it's still A LOT easier than Optical Flow, Depth Recognition, or Robot Navigation... So I'd say: - Text Generation/Transformation: Easy - Speech Recognition: Medium - Optical Flow / Depth Recognition: Hard - Actually controlling robots with AI: Nightmare
neodypsis 4y ago
Regarding the Kaggle competitions, I also agree with that you need access to powerful hardware. Some competitions have very big datasets that require you to have TBs available to download and uncompress the files. Also, having your own GPU will allow you to iterate faster. I tried a competition with just a Colab instance and I felt kinda handicapped, unfortunately.

inciampati 4y ago

Great suggestions and perspective! Thanks!

shoo 4y ago

> to train a state of the art ASR AI, you need roughly 100x A100 for a month, 100,000+ hours of audio recordings, and math knowledge to find a maximum likelihood path through a logit matrix. Do you think there is a distinction between the kinds of problems that take some kind of "raw" signal data (audio, images etc) as input, where deep learning approaches appear to be fruitfully applied, and other kinds of problems that appear in many places in business and the public sector where the input space is not some kind of raw signal data but instead tabular data. I have heard some people argue that the latter kinds of tabular-data style problems can be effectively tackled with a variety of statistical methods, and that deep learning style approaches do not offer an advantage. Maybe fast.ai is commenting on the latter class of problem.

fxtentacle 4y ago
I agree with you :) Most business-style "automate my Excel" problems can be solved pretty well with regular statistical models, meaning you don't need AI to solve them. So my impression is that fast.ai teaches you how to use AI methods, but with example problems that didn't need AI to be solved well.

robga 4y ago

fast.ai openly declares it doesn't have complete pipelines for audio, especially ASR. They're also very open that areas like ASR and RL take a ton of compute to replicate SOTA. This shouldn't be used to substantiate any sort of criticism of fast.ai the library since it is just wrappers on pytorch. Maybe on the course.fast.ai page they could make some sort of disclaimer that some areas like ASR and yet to achieve the quantum leap of performance that we've seen in other domains. I disagree completely that fast.ai is for toy problems. It's a development tool that in my experience provides for more rapid iteration than starting in vanilla pytorch. Use it for toys, for challenges, for research, or in industry. You can always rewrite your solution in pytorch once stable if there is benefit in doing so. I agree with others that fast.ai is a great place to start.

whimsicalism 4y ago

ASR is one of the hardest tasks for a novice to ML/DL to do.

fxtentacle 4y ago
In my opinion, it's still A LOT easier than Optical Flow, Depth Recognition, or Robot Navigation... So I'd say: - Text Generation/Transformation: Easy - Speech Recognition: Medium - Optical Flow / Depth Recognition: Hard - Actually controlling robots with AI: Nightmare

neodypsis 4y ago

Regarding the Kaggle competitions, I also agree with that you need access to powerful hardware. Some competitions have very big datasets that require you to have TBs available to download and uncompress the files. Also, having your own GPU will allow you to iterate faster. I tried a competition with just a Colab instance and I felt kinda handicapped, unfortunately.