4 ms·

What kind of effort does it take to have any idea what you guys are talking about? Is there a For Dummies book?

enjoylife 4y ago
It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg. > # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858 https://arxiv.org/abs/1909.05858) Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.
pksebben 4y ago
there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda. if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense. source: in the middle of doing that right now
- ben_w 4y ago
  > if you troll HN for AI related stuff and just read read read Do you mean "troll" or "trawl"? (I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).
  - Qworg 4y ago
    IIRC, both forms work. Trolling is done by hand with a fishing pole, sometimes on a dock. Trawling is from the back of a boat, generally with a net.
    - ben_w 4y ago
      Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".
      - pksebben 4y ago
        If I'm not hallucinating like GPT, I think the etymology of troll as in internet actually stems from trolling as in fishing. I have a hazy recollection of folks in the early days of 4chan describing 'trolling for idiots' - as in, feigning ignorance to get a rise out of folks. Like, you post a dumb, obviously stupid but infuriating comment and watch people get all worked up about it. You know, for the lulz. I feel like since then it's changed to just actually being a dumb asshole and enjoying when people are mad at you. Or maybe I've just gotten smarter. In any event at the time it certainly didn't seem like the kind of thing racist lunatics would go out of their way to do. Perhaps more the fool I, as I was a teenager at the time and as such that opinion cannot really be trusted.
bootsmann 4y ago
I think the best way is to get a basic understanding of torch and then just reading what the code does. The model internals are hard (though for the transformer there are good step by steps online) but for this case it is basically playing with the output of the actual neural net inside the model. The net gives you a probability distribution to be next over all tokens (think words). Now it has proven to be useful not to pick the token with the highest probability but to employ different strategies. Top-k in this case means probabilistically sampling from the top k tokens instead of piking the best.
sirsinsalot 4y ago
"Transformers for Natural Language Processing", Denis Rothman (Packt)
d0mine 4y ago
To understand how pytorch works (bottom-up with practice video lectures): Neural Networks: Zero to Hero Andrej Karpathy https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...

It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg. > # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858 https://arxiv.org/abs/1909.05858) Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.

pksebben 4y ago

there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda. if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense. source: in the middle of doing that right now

ben_w 4y ago
> if you troll HN for AI related stuff and just read read read Do you mean "troll" or "trawl"? (I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).
- Qworg 4y ago
  IIRC, both forms work. Trolling is done by hand with a fishing pole, sometimes on a dock. Trawling is from the back of a boat, generally with a net.
  - ben_w 4y ago
    Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".
    - pksebben 4y ago
      If I'm not hallucinating like GPT, I think the etymology of troll as in internet actually stems from trolling as in fishing. I have a hazy recollection of folks in the early days of 4chan describing 'trolling for idiots' - as in, feigning ignorance to get a rise out of folks. Like, you post a dumb, obviously stupid but infuriating comment and watch people get all worked up about it. You know, for the lulz. I feel like since then it's changed to just actually being a dumb asshole and enjoying when people are mad at you. Or maybe I've just gotten smarter. In any event at the time it certainly didn't seem like the kind of thing racist lunatics would go out of their way to do. Perhaps more the fool I, as I was a teenager at the time and as such that opinion cannot really be trusted.

bootsmann 4y ago

I think the best way is to get a basic understanding of torch and then just reading what the code does. The model internals are hard (though for the transformer there are good step by steps online) but for this case it is basically playing with the output of the actual neural net inside the model. The net gives you a probability distribution to be next over all tokens (think words). Now it has proven to be useful not to pick the token with the highest probability but to employ different strategies. Top-k in this case means probabilistically sampling from the top k tokens instead of piking the best.

sirsinsalot 4y ago

"Transformers for Natural Language Processing", Denis Rothman (Packt)

d0mine 4y ago

To understand how pytorch works (bottom-up with practice video lectures): Neural Networks: Zero to Hero Andrej Karpathy https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...