7 ms·

Assuming you have the math and algorithmic background, I would start by reading the “attention is all you need” paper. After reading, attempt to build a baby tr

Assuming you have the math and algorithmic background, I would start by reading the “attention is all you need” paper. After reading, attempt to build a baby transformer model in PyTorch. After that, consider constructing some of the building blocks without libraries to understand how they work.

dragonmouse 3y ago
And would you consider this resource to contain "the math and algorithmic background" necessary? Or is it overkill/missing some things? https://www.freecodecamp.org/news/all-the-math-you-need-in-artificial-intelligence/ https://www.freecodecamp.org/news/all-the-math-you-need-in-a...
- Exuma 3y ago
  I'm not qualified to answer this, but I would state nothing is really "overkill" I've been "filling in the gaps" in math for almost a year now to learn machine learning stuff casually. I don't even need to use it, I am just obsessed with learning and I read about it for nearly an hour a day, and its still not enough. Being self-taught at math introduces so many painful problems. If I were to do this seriously I would start ALLLL the way back at algrebra in 5th grade and work forwards ALLL the way up to linear algebra/calculus etc. There's just too many tiny things and subtle details that are missed that I find. It makes any example require 10 times more brain power just to do simple things I don't remember, like the rules of factorization etc. So I'll go learn that thing which is simple, go back and 2 mins later I'm off trying to find some other simple thing. Mainly the idea of learning 5th grade math and such is so boring I just never have actually done it, so really instead of what I do, I would just learn the ENTIRE freaking thing. Learning stuff like gradient descent and stuff is easy, but that's not even where all the hard stuff lies. I feel like trying to understand the math deeply behind those topics where youre not just glossing over explanation is where it gets difficult, and to do that you pretty much need the most solid math background without gaps.
  - bick_nyers 3y ago
    If you have a good handle on undergraduate math material I highly recommend graduate math material. You will "restart" math in a sense and start building up those building blocks piece by piece. If you have a shaky understanding of a derivative that's fine because you will do epsilon-delta proofs until your eyes bleed in a Real Analysis course. Edit: I didn't have the laws/properties of logarithms memorized/understood until maybe 3-4 years into my Math degree. I could have learned it sooner probably, but I just had an aversion to it and would desperately translate any problem into exponentials and work with those instead. I definitely sympathise with the desire to "restart" Math.
  - sn9 3y ago
    > Being self-taught at math introduces so many painful problems. If I were to do this seriously I would start ALLLL the way back at algrebra in 5th grade and work forwards ALLL the way up to linear algebra/calculus etc. If you want to do this, these books are great and have complete solutions manuals available: https://artofproblemsolving.com/store/recommendations https://artofproblemsolving.com/store/recommendations
itake 3y ago
I worry that even if self-taught, I wouldn't have the credentials (job experience or degree), to do a full-time ML job. Our my concerns unfounded?
- tjpnz 3y ago
  ML is a relatively new field. How do you think current engineers got their start?
  - itake 3y ago
    5 years ago, you needed a masters or phd to qualify for a Data Science role. At my current company, that is still the case. Has that change?
    - achileas 3y ago
      DS roles aren't the only roles in the field available.
hcks 3y ago
I read this exact advice often here on HN and I can’t help but wonder. Is the person writing it just repeating something they read? Is it just because they like the ´coding from first principles’ aesthetics? I mean let’s imagine that someone does read that paper, and manage to replicate the code (quite an effort from someone coming from outside AI and academia). Then what? I doubt it’s particularly illuminating. That doesn’t really qualify for a job by itself. So what’s the goal there? Is it just a thing to say to look like a cool hacker that code from scratch?
- lurker919 3y ago
  > That doesn’t really qualify for a job by itself Apple often asks it's ML engineering candidates to explain attention & transformers from scratch.
- nikhizzle 3y ago
  The goal here is to actually understand the mechanics of the model and begin the process of intuitively understanding the space. I would put the effort here at a few weekends of focus. Also will add the models turned out to be a lot simpler to understand than I expected going in.
  - spmurrayzzz 3y ago
    If developing intuitions is the goal, I really do think Jay Alammar's "Illustrated Transformer" [1] is at least a step function better in the service of that outcome than the academic paper itself. (I totally realize this is subjective, but that has been my experience with my own learning in the space over the last few years as well as some folks I've mentored) http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
- janalsncm 3y ago
  I’m an ML engineer and frankly that paper is not very good in my opinion. It’s quite confusing. The BERT paper is much more approachable.
TrackerFF 3y ago
Just a quick reminder to everyone reading that AI / ML (let's face it, it's 0.1% AI and 99.9% ML) is still a ton more than just SOTA Deep Learning models. Depending on where you work, it could be all classical machine learning methods, and zero deep learning - or the other way around. Having a broad enough understanding in ML would be a good starting point, along with solid SW engineering skills.

dragonmouse 3y ago

And would you consider this resource to contain "the math and algorithmic background" necessary? Or is it overkill/missing some things? https://www.freecodecamp.org/news/all-the-math-you-need-in-artificial-intelligence/ https://www.freecodecamp.org/news/all-the-math-you-need-in-a...

Exuma 3y ago
I'm not qualified to answer this, but I would state nothing is really "overkill" I've been "filling in the gaps" in math for almost a year now to learn machine learning stuff casually. I don't even need to use it, I am just obsessed with learning and I read about it for nearly an hour a day, and its still not enough. Being self-taught at math introduces so many painful problems. If I were to do this seriously I would start ALLLL the way back at algrebra in 5th grade and work forwards ALLL the way up to linear algebra/calculus etc. There's just too many tiny things and subtle details that are missed that I find. It makes any example require 10 times more brain power just to do simple things I don't remember, like the rules of factorization etc. So I'll go learn that thing which is simple, go back and 2 mins later I'm off trying to find some other simple thing. Mainly the idea of learning 5th grade math and such is so boring I just never have actually done it, so really instead of what I do, I would just learn the ENTIRE freaking thing. Learning stuff like gradient descent and stuff is easy, but that's not even where all the hard stuff lies. I feel like trying to understand the math deeply behind those topics where youre not just glossing over explanation is where it gets difficult, and to do that you pretty much need the most solid math background without gaps.
- bick_nyers 3y ago
  If you have a good handle on undergraduate math material I highly recommend graduate math material. You will "restart" math in a sense and start building up those building blocks piece by piece. If you have a shaky understanding of a derivative that's fine because you will do epsilon-delta proofs until your eyes bleed in a Real Analysis course. Edit: I didn't have the laws/properties of logarithms memorized/understood until maybe 3-4 years into my Math degree. I could have learned it sooner probably, but I just had an aversion to it and would desperately translate any problem into exponentials and work with those instead. I definitely sympathise with the desire to "restart" Math.
- sn9 3y ago
  > Being self-taught at math introduces so many painful problems. If I were to do this seriously I would start ALLLL the way back at algrebra in 5th grade and work forwards ALLL the way up to linear algebra/calculus etc. If you want to do this, these books are great and have complete solutions manuals available: https://artofproblemsolving.com/store/recommendations https://artofproblemsolving.com/store/recommendations

itake 3y ago

I worry that even if self-taught, I wouldn't have the credentials (job experience or degree), to do a full-time ML job. Our my concerns unfounded?

tjpnz 3y ago
ML is a relatively new field. How do you think current engineers got their start?
- itake 3y ago
  5 years ago, you needed a masters or phd to qualify for a Data Science role. At my current company, that is still the case. Has that change?
  - achileas 3y ago
    DS roles aren't the only roles in the field available.

hcks 3y ago

I read this exact advice often here on HN and I can’t help but wonder. Is the person writing it just repeating something they read? Is it just because they like the ´coding from first principles’ aesthetics? I mean let’s imagine that someone does read that paper, and manage to replicate the code (quite an effort from someone coming from outside AI and academia). Then what? I doubt it’s particularly illuminating. That doesn’t really qualify for a job by itself. So what’s the goal there? Is it just a thing to say to look like a cool hacker that code from scratch?

lurker919 3y ago
> That doesn’t really qualify for a job by itself Apple often asks it's ML engineering candidates to explain attention & transformers from scratch.
nikhizzle 3y ago
The goal here is to actually understand the mechanics of the model and begin the process of intuitively understanding the space. I would put the effort here at a few weekends of focus. Also will add the models turned out to be a lot simpler to understand than I expected going in.
- spmurrayzzz 3y ago
  If developing intuitions is the goal, I really do think Jay Alammar's "Illustrated Transformer" [1] is at least a step function better in the service of that outcome than the academic paper itself. (I totally realize this is subjective, but that has been my experience with my own learning in the space over the last few years as well as some folks I've mentored) http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
janalsncm 3y ago
I’m an ML engineer and frankly that paper is not very good in my opinion. It’s quite confusing. The BERT paper is much more approachable.

TrackerFF 3y ago

Just a quick reminder to everyone reading that AI / ML (let's face it, it's 0.1% AI and 99.9% ML) is still a ton more than just SOTA Deep Learning models. Depending on where you work, it could be all classical machine learning methods, and zero deep learning - or the other way around. Having a broad enough understanding in ML would be a good starting point, along with solid SW engineering skills.