4 ms·

> Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. I was thinking te

> Can you cite the deterministic part? And I don't think fixing your seeds count as making a model not statistical, though yes deterministic. I was thinking temperature=0, but what more do you need than that? (nb there is also some randomness from non-associative FP operations happening in different orders) > I also don't like the term "black box." While they aren't transparent they aren't completely opaque either. A white box is a subset of a black box here. The important part is that the sampler/text generation is wrapped around the transformer model and is not actually the same thing as it.

godelski 2y ago
> I was thinking temperature=0, but what more do you need than that? Temperature isn't a thing exclusive to LLMs or ML. It's a general technique from statistics where you're modifying the variance of the distribution (also see truncation). And as you're aware, temperature zero isn't actually temperature zero. So say such a thing makes something non-statistical is the same as saying prime numbers don't exist because any prime p * 0 is 0. > A white box is a subset of a black box here No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. Complexity isn't what makes something a black box, it's our lack of understanding that does. Of course this tends to coinside, but there's plenty of extremely complex things we have a great understanding of, even if not to infinite precision. Please take your passion and use it to drive you to learn about these things deeper. It's actually the same advice I give to the LLM fanboys who think it's AGI. But trying to dispel that myth without correct information makes it harder. It muddies the waters. These topics are extremely complex and the issue with the ML field is itself a hypocritical claim that it is easy (scale is all you need while calling these machines black boxes). I mean we can talk about the elephant in the room but certainly Von Neumann has said enough.
- astrange 2y ago
  > No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. So it's a subset, because "understanding something" is a refinement of the state of "not understanding something" - you can just pretend you don't know what's happening in that part of the system and it is now a black box.
  - desumeku 2y ago
    Semantic word games aren't a crowd-winning form of argument.
    - godelski 2y ago
      It's worse when you play that semantic game to make your point stronger (changing your original meaning) and willfully misinterpreting the other person. Everyone loves being willfully misinterpreted
    - astrange 2y ago
      I don't think the concept of encapsulation in programming is a semantic word game. Any part of a program can become a black box if you decide it is one (if you can figure out an interface anyway…), and that's good because it means you can swap it out.

godelski 2y ago

> I was thinking temperature=0, but what more do you need than that? Temperature isn't a thing exclusive to LLMs or ML. It's a general technique from statistics where you're modifying the variance of the distribution (also see truncation). And as you're aware, temperature zero isn't actually temperature zero. So say such a thing makes something non-statistical is the same as saying prime numbers don't exist because any prime p * 0 is 0. > A white box is a subset of a black box here No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. Complexity isn't what makes something a black box, it's our lack of understanding that does. Of course this tends to coinside, but there's plenty of extremely complex things we have a great understanding of, even if not to infinite precision. Please take your passion and use it to drive you to learn about these things deeper. It's actually the same advice I give to the LLM fanboys who think it's AGI. But trying to dispel that myth without correct information makes it harder. It muddies the waters. These topics are extremely complex and the issue with the ML field is itself a hypocritical claim that it is easy (scale is all you need while calling these machines black boxes). I mean we can talk about the elephant in the room but certainly Von Neumann has said enough.

astrange 2y ago
> No. A white box and black box sit at opposite ends of a spectrum which is just saying how much we understand something. So it's a subset, because "understanding something" is a refinement of the state of "not understanding something" - you can just pretend you don't know what's happening in that part of the system and it is now a black box.
- desumeku 2y ago
  Semantic word games aren't a crowd-winning form of argument.
  - godelski 2y ago
    It's worse when you play that semantic game to make your point stronger (changing your original meaning) and willfully misinterpreting the other person. Everyone loves being willfully misinterpreted
  - astrange 2y ago
    I don't think the concept of encapsulation in programming is a semantic word game. Any part of a program can become a black box if you decide it is one (if you can figure out an interface anyway…), and that's good because it means you can swap it out.