5 ms·

I wonder how true "zero shot" is after you fed them basically all the text on the internet. To me, that sounds a bit like saying "this mathematics PhD mastered

I wonder how true "zero shot" is after you fed them basically all the text on the internet. To me, that sounds a bit like saying "this mathematics PhD mastered differentiating x3+x2 on the first try!!!".

chpatrick 3y ago
Well the whole internet doesn't fit into the few gigabytes these models are.
- benreesman 3y ago
  Few gigabytes? The smallest ones quantized to where you have to squint a little. 70B weights at fp32 is a lot of Wikipedia. GPT-4 is rumored to be 1.5 terraweights. There’s compression but it’s not magical.
  - chpatrick 3y ago
    Even 6TB is a couple garden variety hard drives, that's not much compared to the internet.
    - usrnm 3y ago
      What if "the internet" is just the same stuff under slightly dufferent dressings, and LLMs are good at seeing through it?
      - chpatrick 3y ago
        That's exactly my point.
      - notnaut 3y ago
        Well, yeah. In the same sense that all songs are just the same stuff dressed differently because there are finite musical notes. Doesn’t really take away from the majesty of music.
- Salgat 3y ago
  English wikipedia text uncompressed is 60GB, and that's 6.7 million articles. You can fit an ungodly amount of information in less than a terabyte.