4 ms·

I read this exact advice often here on HN and I can’t help but wonder. Is the person writing it just repeating something they read? Is it just because they lik

by hcks 3y ago

I read this exact advice often here on HN and I can’t help but wonder. Is the person writing it just repeating something they read? Is it just because they like the ´coding from first principles’ aesthetics? I mean let’s imagine that someone does read that paper, and manage to replicate the code (quite an effort from someone coming from outside AI and academia). Then what? I doubt it’s particularly illuminating. That doesn’t really qualify for a job by itself. So what’s the goal there? Is it just a thing to say to look like a cool hacker that code from scratch?

lurker919 3y ago
> That doesn’t really qualify for a job by itself Apple often asks it's ML engineering candidates to explain attention & transformers from scratch.
nikhizzle 3y ago
The goal here is to actually understand the mechanics of the model and begin the process of intuitively understanding the space. I would put the effort here at a few weekends of focus. Also will add the models turned out to be a lot simpler to understand than I expected going in.
- spmurrayzzz 3y ago
  If developing intuitions is the goal, I really do think Jay Alammar's "Illustrated Transformer" [1] is at least a step function better in the service of that outcome than the academic paper itself. (I totally realize this is subjective, but that has been my experience with my own learning in the space over the last few years as well as some folks I've mentored) http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/
janalsncm 3y ago
I’m an ML engineer and frankly that paper is not very good in my opinion. It’s quite confusing. The BERT paper is much more approachable.
- areyoukillingm3 3y ago
  Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805
mathisfun123 3y ago
> Is the person writing it just repeating something they read? Is it just because they like the ´coding from first principles’ aesthetics? of course. you know how i know? absolutely no one except the wannabees has time to read papers - people working in the area have deadlines and meetings. we absorb the content of the paper by osmosis - convos, code bases, occasionally a talk at a conference. it's especially horrible advice from the perspective of pedagogy to tell a n00b to read a paper (so the person giving the advice has immediately disqualified themselves from possibly being an expert) because papers are horribly written, omit critical details, and function purely as advertising for the authors, group, etc. for every poor undergrad/n00b soul reading this comment, take this thing to heart that took me too long to unlearn (due to its constant perpetuation by people like gop): reading the paper is 100:1 waste-of-time:value-derived. if i hear about/see some paper that makes strong claims that are relevant to my work, i will look for a github link and/or email the first author. 5/10 i get a response (ratio is going up as i'm getting to be more ingratiated in my community). the other times i just move on - none of these papers have some revolutionary cure for cancer in them so most of the time what i'm already doing is already close enough that i don't need to kill myself figuring out the new thing. that paper in particular (attention) has nothing in it that is in the least bit interesting/revolutionary. the hard part of attention isn't writing down softmax(QK^T)V, the hard part is executing that matrix product fast enough that you're not waiting eons for your model to converge.
thebigwinning 3y ago
> I doubt it’s particularly illuminating. Everytime I do a foundational activity like this it does turn out to be illuminating. Why do you doubt you'll learn something?

by hcks 3y ago

lurker919 3y ago

> That doesn’t really qualify for a job by itself Apple often asks it's ML engineering candidates to explain attention & transformers from scratch.

nikhizzle 3y ago

The goal here is to actually understand the mechanics of the model and begin the process of intuitively understanding the space. I would put the effort here at a few weekends of focus. Also will add the models turned out to be a lot simpler to understand than I expected going in.

spmurrayzzz 3y ago
If developing intuitions is the goal, I really do think Jay Alammar's "Illustrated Transformer" [1] is at least a step function better in the service of that outcome than the academic paper itself. (I totally realize this is subjective, but that has been my experience with my own learning in the space over the last few years as well as some folks I've mentored) http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/

janalsncm 3y ago

I’m an ML engineer and frankly that paper is not very good in my opinion. It’s quite confusing. The BERT paper is much more approachable.

areyoukillingm3 3y ago
Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805

mathisfun123 3y ago

> Is the person writing it just repeating something they read? Is it just because they like the ´coding from first principles’ aesthetics? of course. you know how i know? absolutely no one except the wannabees has time to read papers - people working in the area have deadlines and meetings. we absorb the content of the paper by osmosis - convos, code bases, occasionally a talk at a conference. it's especially horrible advice from the perspective of pedagogy to tell a n00b to read a paper (so the person giving the advice has immediately disqualified themselves from possibly being an expert) because papers are horribly written, omit critical details, and function purely as advertising for the authors, group, etc. for every poor undergrad/n00b soul reading this comment, take this thing to heart that took me too long to unlearn (due to its constant perpetuation by people like gop): reading the paper is 100:1 waste-of-time:value-derived. if i hear about/see some paper that makes strong claims that are relevant to my work, i will look for a github link and/or email the first author. 5/10 i get a response (ratio is going up as i'm getting to be more ingratiated in my community). the other times i just move on - none of these papers have some revolutionary cure for cancer in them so most of the time what i'm already doing is already close enough that i don't need to kill myself figuring out the new thing. that paper in particular (attention) has nothing in it that is in the least bit interesting/revolutionary. the hard part of attention isn't writing down softmax(QK^T)V, the hard part is executing that matrix product fast enough that you're not waiting eons for your model to converge.

thebigwinning 3y ago

> I doubt it’s particularly illuminating. Everytime I do a foundational activity like this it does turn out to be illuminating. Why do you doubt you'll learn something?