4 ms·

I love this, something that starts off as some kind of sorcery a year ago is now being explained so well and almost in a childish way.

maelito 3y ago
> and almost in a childish way. No. You've got to have a solid background in computer science to even start to understand fully this article. Even the title itself is not accessible to 99% of humans.
- bookofjoe 3y ago
  Count me in as one of the 99%
- coldtea 3y ago
  Parent obviously means childish as in "simple and easy manner" not as "something 10 year olds will understand".
Hendrikto 3y ago
[flagged]
- worewood 3y ago
  Also, about the non-determinism issue, there was a post some time ago and that comes from the way the GPU does the calculations, something something floating point something. So of course the algorithm is deterministic, but the real-life implementation isn't.
  - gugagore 3y ago
    Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result. But there is probabilistic sampling that happens (see "temperature").
    - marginalia_nu 3y ago
      > Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result. In this sense, I don't think it's fair to say floating point math is non-deterministic, as much as parallel computation is non-deterministic. FP behaves in unexpected ways, but the same order of operations always yields the same unexpected results (except on Pentium 1).
  - deleted 3y ago
    [deleted]
- dzink 3y ago
  Electricity, Cars, and Gas were once upon a time a luxury as well - reserved to those who could afford them or had unique access / credentials / skills. The people who were able to simplify and spread the advanced tech to the common person became Billionaires.
- darkmx0z 3y ago
  when talking about a set of characters, alphabet is a commonly used term https://cs.stackexchange.com/search?q=string++alphabet+size https://cs.stackexchange.com/search?q=string++alphabet+size
  - ecnahc515 3y ago
    Sure, but if your doing work in machine learning that’s generally not the terminology used, hinting that this isn’t the area the author specializes in (which isn’t a bad thing, but take their explanations with a grain of salt).
pedrosorio 3y ago
This sorcery didn't start a year ago. The model being described in the article is GPT-2 which was released in early 2019.

maelito 3y ago

> and almost in a childish way. No. You've got to have a solid background in computer science to even start to understand fully this article. Even the title itself is not accessible to 99% of humans.

bookofjoe 3y ago
Count me in as one of the 99%
coldtea 3y ago
Parent obviously means childish as in "simple and easy manner" not as "something 10 year olds will understand".

Hendrikto 3y ago

[flagged]

worewood 3y ago
Also, about the non-determinism issue, there was a post some time ago and that comes from the way the GPU does the calculations, something something floating point something. So of course the algorithm is deterministic, but the real-life implementation isn't.
- gugagore 3y ago
  Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result. But there is probabilistic sampling that happens (see "temperature").
  - marginalia_nu 3y ago
    > Floating point addition, for example, is not associative, so the order of taking a sum affects the result. If the summation were sequential and single threaded, it would be deterministic. But it happens in parallel, so timing variations affect the result. In this sense, I don't think it's fair to say floating point math is non-deterministic, as much as parallel computation is non-deterministic. FP behaves in unexpected ways, but the same order of operations always yields the same unexpected results (except on Pentium 1).
- deleted 3y ago
  [deleted]
dzink 3y ago
Electricity, Cars, and Gas were once upon a time a luxury as well - reserved to those who could afford them or had unique access / credentials / skills. The people who were able to simplify and spread the advanced tech to the common person became Billionaires.
darkmx0z 3y ago
when talking about a set of characters, alphabet is a commonly used term https://cs.stackexchange.com/search?q=string++alphabet+size https://cs.stackexchange.com/search?q=string++alphabet+size
- ecnahc515 3y ago
  Sure, but if your doing work in machine learning that’s generally not the terminology used, hinting that this isn’t the area the author specializes in (which isn’t a bad thing, but take their explanations with a grain of salt).

pedrosorio 3y ago

This sorcery didn't start a year ago. The model being described in the article is GPT-2 which was released in early 2019.