3 ms·
> Fundamentally, it still "sees" the world through text Fundamentally, it "sees the world" [0] through tokens, which are not text. [0] Also a bad metaphor, bu
by dragonwriter 3y ago
> Fundamentally, it still "sees" the world through text
Fundamentally, it "sees the world" [0] through tokens, which are not text.
[0] Also a bad metaphor, but...
- int_19h 3y agoTechnically true, but when those tokens are 1:1 mapped to text, I think we can simplify this down without losing anything important. Of course, once you start using tokens for other things - as multimodal LMs already do - that changes. But this current crop of model still has visual modality in its infancy IMO, and gauging the overall performance of model as a whole based on that is very questionable.
- dragonwriter 3y ago> Technically true, but when those tokens are 1:1 mapped to text I don't know what GPT-4V does particularly, but my understanding is that multimodal models very often have an expanded token space with special tokens related to image handling, so, literally, there is not a 1:1 relationship of tokens to text.
- Jensson 3y agoA string of tokens is text. Tokens is just another alphabet, like Japanese letters having many representations for the same sounds and a letter can be entire words sometimes.