5 ms·

The Engineer’s Guide to Deep Learning: Understanding the Transformer Model

258 points by shxx 2y ago

deleted 2y ago
[deleted]
gregw2 2y ago
No content besides a few paragraphs of intro. Actual content has 404 not found errors.
- deleted 2y ago
  [deleted]
- gpnt 2y ago
  The menu on the left on desktop is working. It seems only the links on the first page are broken.
- smokel 2y ago
  The links in the running text are broken. The links in the hamburger menu work fine.
_giorgio_ 2y ago
It uses keras, which is obsolete. Nobody uses that thing anymore. Stay away from this.
- 0xd1r 2y ago
  I suppose they are using keras because of its simplicity. But I agree, things have really moved on from keras.
- albertzeyer 2y ago
  I wonder about Keras 3. It's now backend independent again, like in the early days, and supports JAX, TensorFlow, or PyTorch. It's a nice thing if you defined your model and can then easily switch between the frameworks, right? Or no-one cares about that, and everyone just uses PyTorch?
- sva_ 2y ago
  I prefer PyTorch myself, but to call Keras obsolete is quite the stretch. Just because academia has largely moved on from it, doesn't mean nobody uses it. Also, the API isn't all that different from other libraries. The principles are the same.
  - whiplash451 2y ago
    The industry has also moved on from keras/tensorflow (apart from for legacy reasons). Google itself has moved on to JAX.
    - deleted 2y ago
      [deleted]
  - _giorgio_ 2y ago
    [flagged]
tuyguntn 2y ago
question to experts of HN in ML/AI. Could you please share the beginner resources you think would worth for a person who wants to switch their domain from CRUD/backend APIs to ML/AI. There seems to be many branches of this domain, not sure where to start. Is my understanding correct? * ML engineer -> engineer who builds ML models with pytorch (or similar frameworks) * AI engineer -> engineer who builds applications on top of AI solutions (prompt engineering, OpenAI, Claude APIs,....) * ML ops -> people who help with deploying, serving models
- qsort 2y ago
  None of these terms have a formal definition. The only association rule you need is: * Fancy Title -> Whatever the company wants it to be. All of the above could realistically span from "does bleeding-edge work" to "has once opened a CSV".
- treme 2y ago
  Kaggle is a good start
- blowski 2y ago
  I'd call the "AI Engineer" an Application Engineer, albeit one that specialises in integrating ML into software.
- belter 2y ago
  85% of your ML project time will be spent on Data Quality and a little bit of Domain Feature Engineering. If you want to make an impact, become excellent at those, you will be able to use these skills, for domains like Systems Integration and Business Analytics. Let the people who do Research bring you the Algorithms and nowadays even the trained Models.
shubham13596 2y ago
Very good resource on building from the basics in a concise manner.
MAXPOOL 2y ago
There are many others that are better. 1/ The Annotated Transformer Attention is All You Need http://nlp.seas.harvard.edu/annotated-transformer/ http://nlp.seas.harvard.edu/annotated-transformer/ 2/ Transformers from Scratch https://e2eml.school/transformers.html https://e2eml.school/transformers.html 3/ Andrej Karpathy has really good series of intros: https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY GPT with Andrej Karpathy: Part 1 https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1-865bec6fbcce https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1... 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning https://www.youtube.com/watch?v=wjZofJX0v4M https://www.youtube.com/watch?v=wjZofJX0v4M Attention in transformers, visually explained | Chapter 6, Deep Learning https://www.youtube.com/watch?v=eMlx5fFNoYc https://www.youtube.com/watch?v=eMlx5fFNoYc Full 3Blue1Brown Neural Networks playlist https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...
- rvnx 2y ago
  In addition, these websites are totally free. The website listed here: > I consider requests for full commercial use of all content on this site (and the github repository). For a complete buyout of all content rights, the cost is €10,000,000. > I’d like to ask you what problems you have by that I keep on having the copyright of my document. + no commercial-use without paying 20% royalty. So fairly expensive for a Keras tutorial.
  - bitmaks 2y ago
    I think that's pretty obviously a joke, no?
- mr_puzzled 2y ago
  Slightly off topic: I'm interested in taking part in the Vesuvius challenge[0], but I don't have a background in ML, just a regular web developer. Does anyone have suggestions on how to get started? I planned to get some background on practical ML by working through Karpathy's Zero to Hero series along with the Understanding Deep Learning book. Would that be enough or anything else I should learn? I plan to understand the existing solutions to last year's prize and then pick a smaller sub challenge. [0] https://scrollprize.org/ https://scrollprize.org/
alister 2y ago
> When you send me an email, please provide at least two SNS [social networking service] addresses (e.g. LinkedIn, Twitter) for verification purposes. ... I no longer accept contact from anonymous individuals. It's pretty sad to see that social networking is being adopted as an identification and trust mechanism even by technical people. It was bad enough when some governments began demanding social networking usernames for visa/immigrant screening, but we can't even send an email without social proof to other technical people now?
- belter 2y ago
  > I no longer accept contact from anonymous individuals. This reminds of that joke, where a guy shows up at the Air Force HQ recruitment center. They ask, "Pilot license? Experience? Qualifications?" He replies, "Nope, just here to say: Don't count on me!"
- kltzayh 2y ago
  With the ellipsis expanded: "Due to the XZ backdoor incident, I no longer accept contact from anonymous individuals." The XZ cracker could have logged in via GitHub at numerous services. I bet that the OP downloads from PyPI that was potentially compromised for longer than a year due to an overlooked token leak. I further bet that the OP, being in the machine learning space, downloads unauditable, huge Python frameworks from GitHub, conda or PyPI. People in that space also download and experiment with untrusted models. But hey, plain text email which you can read in a command line mail client with MIME and other extensions disabled is the problem!
revskill 2y ago
Transformer tutorial is like the new "Monad tutorial".
benterix 2y ago
> In contrast, the AI technology of the current golden age, which began in the mid-2010s, has consistently exceeded our expectations. Well, until recently, that is. It looks like we hit the wall as for what LLMs can do - some might call it a plateau of productivity. Namely, as far as coding is concerned, LLMs can successfully create chunk of code of limited length and tiny programs, can also review small pieces of code and suggest improvements that are not related to the context of the whole program (unless it can fit in the context window). In spite of huge effort put in creating a system where LLM agents could work together to create software such as AutoGPT, no non-trivial program has been created in this way so far.
yobbo 2y ago
This is a very compressed work-through from perceptron to transformer. When he is working through the gradients of an LSTM, for example, it is to help understanding, not help you implement it in your favourite framework. When he is showing solutions in various frameworks, the purpose is to help create connections between what the math looks like and what code can look like.
uoaei 2y ago
One of the most frustrating things about all the documentation on Transformers is the sole emphasis on NLP. In particular, one of the most interesting parts of the Transformer architecture to me is the attention mechanism which is permutation invariant (if not for the positional embeddings people use to counteract this inherent quality of attention layers). Also the ability to arbitrarily mask this or that node in the graph -- or even individual edges -- gives the whole thing so much flexibility for encoding domain knowledge into your architecture. Positional embeddings may still be required in many cases but you can be clever about them beyond the overly restrictive perspective of attention layer inputs purely as one-dimensional sequences.
jhkug 2y ago
[flagged]

258 points by shxx 2y ago

deleted 2y ago

[deleted]

gregw2 2y ago

No content besides a few paragraphs of intro. Actual content has 404 not found errors.

deleted 2y ago
[deleted]
gpnt 2y ago
The menu on the left on desktop is working. It seems only the links on the first page are broken.
smokel 2y ago
The links in the running text are broken. The links in the hamburger menu work fine.

_giorgio_ 2y ago

It uses keras, which is obsolete. Nobody uses that thing anymore. Stay away from this.

0xd1r 2y ago
I suppose they are using keras because of its simplicity. But I agree, things have really moved on from keras.
albertzeyer 2y ago
I wonder about Keras 3. It's now backend independent again, like in the early days, and supports JAX, TensorFlow, or PyTorch. It's a nice thing if you defined your model and can then easily switch between the frameworks, right? Or no-one cares about that, and everyone just uses PyTorch?
sva_ 2y ago
I prefer PyTorch myself, but to call Keras obsolete is quite the stretch. Just because academia has largely moved on from it, doesn't mean nobody uses it. Also, the API isn't all that different from other libraries. The principles are the same.
- whiplash451 2y ago
  The industry has also moved on from keras/tensorflow (apart from for legacy reasons). Google itself has moved on to JAX.
  - deleted 2y ago
    [deleted]
- _giorgio_ 2y ago
  [flagged]

tuyguntn 2y ago

question to experts of HN in ML/AI. Could you please share the beginner resources you think would worth for a person who wants to switch their domain from CRUD/backend APIs to ML/AI. There seems to be many branches of this domain, not sure where to start. Is my understanding correct? * ML engineer -> engineer who builds ML models with pytorch (or similar frameworks) * AI engineer -> engineer who builds applications on top of AI solutions (prompt engineering, OpenAI, Claude APIs,....) * ML ops -> people who help with deploying, serving models

qsort 2y ago
None of these terms have a formal definition. The only association rule you need is: * Fancy Title -> Whatever the company wants it to be. All of the above could realistically span from "does bleeding-edge work" to "has once opened a CSV".
treme 2y ago
Kaggle is a good start
blowski 2y ago
I'd call the "AI Engineer" an Application Engineer, albeit one that specialises in integrating ML into software.
belter 2y ago
85% of your ML project time will be spent on Data Quality and a little bit of Domain Feature Engineering. If you want to make an impact, become excellent at those, you will be able to use these skills, for domains like Systems Integration and Business Analytics. Let the people who do Research bring you the Algorithms and nowadays even the trained Models.

shubham13596 2y ago

Very good resource on building from the basics in a concise manner.

MAXPOOL 2y ago

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need http://nlp.seas.harvard.edu/annotated-transformer/ http://nlp.seas.harvard.edu/annotated-transformer/ 2/ Transformers from Scratch https://e2eml.school/transformers.html https://e2eml.school/transformers.html 3/ Andrej Karpathy has really good series of intros: https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY GPT with Andrej Karpathy: Part 1 https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1-865bec6fbcce https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1... 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning https://www.youtube.com/watch?v=wjZofJX0v4M https://www.youtube.com/watch?v=wjZofJX0v4M Attention in transformers, visually explained | Chapter 6, Deep Learning https://www.youtube.com/watch?v=eMlx5fFNoYc https://www.youtube.com/watch?v=eMlx5fFNoYc Full 3Blue1Brown Neural Networks playlist https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...

rvnx 2y ago
In addition, these websites are totally free. The website listed here: > I consider requests for full commercial use of all content on this site (and the github repository). For a complete buyout of all content rights, the cost is €10,000,000. > I’d like to ask you what problems you have by that I keep on having the copyright of my document. + no commercial-use without paying 20% royalty. So fairly expensive for a Keras tutorial.
- bitmaks 2y ago
  I think that's pretty obviously a joke, no?
mr_puzzled 2y ago
Slightly off topic: I'm interested in taking part in the Vesuvius challenge[0], but I don't have a background in ML, just a regular web developer. Does anyone have suggestions on how to get started? I planned to get some background on practical ML by working through Karpathy's Zero to Hero series along with the Understanding Deep Learning book. Would that be enough or anything else I should learn? I plan to understand the existing solutions to last year's prize and then pick a smaller sub challenge. [0] https://scrollprize.org/ https://scrollprize.org/

alister 2y ago

> When you send me an email, please provide at least two SNS [social networking service] addresses (e.g. LinkedIn, Twitter) for verification purposes. ... I no longer accept contact from anonymous individuals. It's pretty sad to see that social networking is being adopted as an identification and trust mechanism even by technical people. It was bad enough when some governments began demanding social networking usernames for visa/immigrant screening, but we can't even send an email without social proof to other technical people now?

belter 2y ago
> I no longer accept contact from anonymous individuals. This reminds of that joke, where a guy shows up at the Air Force HQ recruitment center. They ask, "Pilot license? Experience? Qualifications?" He replies, "Nope, just here to say: Don't count on me!"
kltzayh 2y ago
With the ellipsis expanded: "Due to the XZ backdoor incident, I no longer accept contact from anonymous individuals." The XZ cracker could have logged in via GitHub at numerous services. I bet that the OP downloads from PyPI that was potentially compromised for longer than a year due to an overlooked token leak. I further bet that the OP, being in the machine learning space, downloads unauditable, huge Python frameworks from GitHub, conda or PyPI. People in that space also download and experiment with untrusted models. But hey, plain text email which you can read in a command line mail client with MIME and other extensions disabled is the problem!

revskill 2y ago

Transformer tutorial is like the new "Monad tutorial".

benterix 2y ago

> In contrast, the AI technology of the current golden age, which began in the mid-2010s, has consistently exceeded our expectations. Well, until recently, that is. It looks like we hit the wall as for what LLMs can do - some might call it a plateau of productivity. Namely, as far as coding is concerned, LLMs can successfully create chunk of code of limited length and tiny programs, can also review small pieces of code and suggest improvements that are not related to the context of the whole program (unless it can fit in the context window). In spite of huge effort put in creating a system where LLM agents could work together to create software such as AutoGPT, no non-trivial program has been created in this way so far.

yobbo 2y ago

This is a very compressed work-through from perceptron to transformer. When he is working through the gradients of an LSTM, for example, it is to help understanding, not help you implement it in your favourite framework. When he is showing solutions in various frameworks, the purpose is to help create connections between what the math looks like and what code can look like.

uoaei 2y ago

One of the most frustrating things about all the documentation on Transformers is the sole emphasis on NLP. In particular, one of the most interesting parts of the Transformer architecture to me is the attention mechanism which is permutation invariant (if not for the positional embeddings people use to counteract this inherent quality of attention layers). Also the ability to arbitrarily mask this or that node in the graph -- or even individual edges -- gives the whole thing so much flexibility for encoding domain knowledge into your architecture. Positional embeddings may still be required in many cases but you can be clever about them beyond the overly restrictive perspective of attention layer inputs purely as one-dimensional sequences.

jhkug 2y ago

[flagged]