3 ms·

Wikipedia is about 24GB, so if you're allowed to drop 1/3 of the details and make up the missing parts by splicing in random text, 8GB doesn't sound too bad. T

by exe34 1y ago

Wikipedia is about 24GB, so if you're allowed to drop 1/3 of the details and make up the missing parts by splicing in random text, 8GB doesn't sound too bad. To me the amazing thing is that you can tell the model to do something, even follow simple instructions in plain English, like make a list or write some python code to do $x, that's the really amazing part.

b112 1y ago
Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668 https://arxiv.org/pdf/2309.10668 So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable. It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.
- horsawlarway 1y ago
  The 24G is the compressed number. They host a pretty decent article here: https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia The relevant bit: > As of 16 October 2024, the size of the current version including all articles compressed is about 24.05 GB without media.
  - b112 1y ago
    Nice link, thanks. Well I'll fallback position, and say one is lossy, the other not.
Nevermark 1y ago
It blows my mind that I can ask for 50 synonyms, instantly get a great list with great meaning summaries. Then ask for the same list sorted and get that nearly instantly, These models have a short time context for now, but they already have a huge “working memory” relative to us. It is very cool. And indicative that vastly smarter models are going to be achieved fairly easily, with new insight. Our biology has had to ruthlessly work within our biological/ecosystem energy envelope, and with the limited value/effort returned by a pre-internet pre-vast economy. So biology has never been able to scale. Just get marginally more efficient and effective within tight limits. Suddenly, (in historical, biological terms), energy availability limits have been removed, and limits on the value of work have compounded and continue to do so. Unsurprising that those changes suddenly unlock easily achieved vast untapped room for cognitive upscaling.
- Wowfunhappy 1y ago
  > These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior working memories to compensate for their limited reasoning abilities. This can make them very useful tools! But there may well be a ceiling to how far that can go. When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportunities to draw on its huge memory bank and try to put things together. But humans are able to reason with orders of magnitude less training data. And by the way, we are out of new training data to give the models.
  - antonvs 1y ago
    > Have much more limited logical reasoning skills. Relative to the best humans, perhaps, but I seriously doubt this is true in general. Most people I work with couldn’t reason nearly as well through the questions I use LLMs to answer. It’s also worth keeping in mind that having a different approach to reasoning is not necessarily equivalent to a worse approach. Watch out for cherry-picking the cons of its approach and ignoring the pros.
    - exe34 1y ago
      > Relative to the best humans, For some reason, the bar for AI is always against the best possible human, right now.
      - antonvs 1y ago
        It seems that 90% of discussion about AI boils down to people who feel threatened by it in some way, and are lashing out in irrational ways as a result. (Source for 90% figure: Sturgeon's Law.)

by exe34 1y ago

b112 1y ago

Not to mention, Language Modeling is Compression https://arxiv.org/pdf/2309.10668 https://arxiv.org/pdf/2309.10668 So text wikipedia at 24G would easily hit 8G with many standard forms of compression, I'd think. If not better. And it would be 100% accurate, full text and data. Far more usable. It's so easy for people to not realise how massive 8GB really is, in terms of text. Especially if you use ascii instead of UTF.

horsawlarway 1y ago
The 24G is the compressed number. They host a pretty decent article here: https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia The relevant bit: > As of 16 October 2024, the size of the current version including all articles compressed is about 24.05 GB without media.
- b112 1y ago
  Nice link, thanks. Well I'll fallback position, and say one is lossy, the other not.

Nevermark 1y ago

It blows my mind that I can ask for 50 synonyms, instantly get a great list with great meaning summaries. Then ask for the same list sorted and get that nearly instantly, These models have a short time context for now, but they already have a huge “working memory” relative to us. It is very cool. And indicative that vastly smarter models are going to be achieved fairly easily, with new insight. Our biology has had to ruthlessly work within our biological/ecosystem energy envelope, and with the limited value/effort returned by a pre-internet pre-vast economy. So biology has never been able to scale. Just get marginally more efficient and effective within tight limits. Suddenly, (in historical, biological terms), energy availability limits have been removed, and limits on the value of work have compounded and continue to do so. Unsurprising that those changes suddenly unlock easily achieved vast untapped room for cognitive upscaling.

Wowfunhappy 1y ago
> These models [...] have a huge “working memory” relative to us. [This is] indicative that vastly smarter models are going to be achieved fairly easily, with new insight. I don't think your second sentence logically follows from the first. Relative to us, these models: - Have a much larger working memory. - Have much more limited logical reasoning skills. To some extent, these models are able to use their superior working memories to compensate for their limited reasoning abilities. This can make them very useful tools! But there may well be a ceiling to how far that can go. When you ask a model to "think about the problem step by step" to improve its reasoning, you are basically just giving it more opportunities to draw on its huge memory bank and try to put things together. But humans are able to reason with orders of magnitude less training data. And by the way, we are out of new training data to give the models.
- antonvs 1y ago
  > Have much more limited logical reasoning skills. Relative to the best humans, perhaps, but I seriously doubt this is true in general. Most people I work with couldn’t reason nearly as well through the questions I use LLMs to answer. It’s also worth keeping in mind that having a different approach to reasoning is not necessarily equivalent to a worse approach. Watch out for cherry-picking the cons of its approach and ignoring the pros.
  - exe34 1y ago
    > Relative to the best humans, For some reason, the bar for AI is always against the best possible human, right now.
    - antonvs 1y ago
      It seems that 90% of discussion about AI boils down to people who feel threatened by it in some way, and are lashing out in irrational ways as a result. (Source for 90% figure: Sturgeon's Law.)