3 ms·

When I saw "from scratch" instead of "pretrained" I assumed this was trained using some novel RL setup-- there's value to using the same verbiage as everyone el

When I saw "from scratch" instead of "pretrained" I assumed this was trained using some novel RL setup-- there's value to using the same verbiage as everyone else.

leoff 3y ago
It's scratch, the programming language https://scratch.mit.edu/about https://scratch.mit.edu/about
zapnuk 3y ago
These pedantic “when I saw X instead of Y I assumed Z” are the most annoying comments on this website. “From scratch” might be non-specific, but a fine phrasing in this case.
- huggingmouth 3y ago
  No, they're not padentic or "anoying". Words have meaning. The more precise their use is, the clearer the communication becomes. Investing the time to properly learn pre-established termilology has tremendous value for everyone. Well, everyone but those who are too lazy and sloppy, and would rather just confuse everybody instead. I'm glad this disgusting sloppiness doesn't exist in more established fields. I hope people keep calling it out.
  - nineteen999 3y ago
    > "disgusting sloppiness" > padentic > anoying > termilology Cute.
    - huggingmouth 3y ago
      Cute or not. My point stands
- popinman322 3y ago
  It's valuable data sometimes. Those types of comments show you how other people perceived something. In this case my lens is colored by my interests; I've been thinking a lot on RL for LLMs and it slipped out here. For me I rely on phrases to quickly filter for (subjectively) interesting work. And assuming we value that subjective filtering of content, we make it easier for others to filter consistently when we use consistent language.
synarchefriend 3y ago
They probably want to emphasize that it's not another llama derivative.
Mougatine 3y ago
"From scratch" is commonly used in the field.
- popinman322 3y ago
  Among arxiv publications there are 217 results that contain "large language model" in the full text and "from scratch" in the title or abstract. There are 2873 results that contain "large language model" in the full text and use "pretrained" in the title or abstract. A 10x difference in publication count does make one feel more common than the other? I'd need to get into more involved queries to break down the semantic categories of those papers.
  - light_hue_1 3y ago
    From scratch simply means that they didn't base it off some other llm. This is perfectly good language and exactly the correct thing for them to say.
GaggiX 3y ago
"Trained from scratch" is perfectly fine terminology to report that the model they published is not a finetuned, but was trained from randomly initialized weights.

When I saw "from scratch" instead of "pretrained" I assumed this was trained using some novel RL setup-- there's value to using the same verbiage as everyone else.

leoff 3y ago

It's scratch, the programming language https://scratch.mit.edu/about https://scratch.mit.edu/about

zapnuk 3y ago

These pedantic “when I saw X instead of Y I assumed Z” are the most annoying comments on this website. “From scratch” might be non-specific, but a fine phrasing in this case.

huggingmouth 3y ago
No, they're not padentic or "anoying". Words have meaning. The more precise their use is, the clearer the communication becomes. Investing the time to properly learn pre-established termilology has tremendous value for everyone. Well, everyone but those who are too lazy and sloppy, and would rather just confuse everybody instead. I'm glad this disgusting sloppiness doesn't exist in more established fields. I hope people keep calling it out.
- nineteen999 3y ago
  > "disgusting sloppiness" > padentic > anoying > termilology Cute.
  - huggingmouth 3y ago
    Cute or not. My point stands
popinman322 3y ago
It's valuable data sometimes. Those types of comments show you how other people perceived something. In this case my lens is colored by my interests; I've been thinking a lot on RL for LLMs and it slipped out here. For me I rely on phrases to quickly filter for (subjectively) interesting work. And assuming we value that subjective filtering of content, we make it easier for others to filter consistently when we use consistent language.

synarchefriend 3y ago

They probably want to emphasize that it's not another llama derivative.

Mougatine 3y ago

"From scratch" is commonly used in the field.

popinman322 3y ago
Among arxiv publications there are 217 results that contain "large language model" in the full text and "from scratch" in the title or abstract. There are 2873 results that contain "large language model" in the full text and use "pretrained" in the title or abstract. A 10x difference in publication count does make one feel more common than the other? I'd need to get into more involved queries to break down the semantic categories of those papers.
- light_hue_1 3y ago
  From scratch simply means that they didn't base it off some other llm. This is perfectly good language and exactly the correct thing for them to say.

GaggiX 3y ago

"Trained from scratch" is perfectly fine terminology to report that the model they published is not a finetuned, but was trained from randomly initialized weights.