3 ms·

Robert <|endoftext|> Tables strikes again!

• deleted 4y ago
  [deleted]
• jetrink 4y ago
  I'm surprised we're making this mistake again. Isn't it possible to structure the chats as something more than a flat stream of text and insert unrepresentable tokens after the tokenization step?
  • arthurcolle 4y ago
    It is almost like they are making it deliberately easy to play with/jailbreak!
    • jahsome 4y ago
      The cynic in me has at times suspected the openai and Microsoft marketing teams are skipping hand in hand all the way to the bank singing "no such thing as bad press."
  • Spivak 4y ago
    Yes and no. You can represent chats as structured data but you have to leave the airlock eventually and render it to flat text because the model only knows "complete the next word" and that in-band structure is context for the model itself to understand the flow of messages. So the model is seeing a series of <|start|>text<|end|> and using that to figure out who's talking but if you convince the model to output <|end|><|start|> then when it goes to complete the next token it has no way to tell the difference.
    • wongarsu 4y ago
      From the point of view of the model, there is no text, only a stream of tokens. There is no need for all tokens to map to valid ascii or unicode. You could have an arbitrary number of tokens for control flow that mean something to the software, but don't have a canonical way to encode in text.
  • Dalewyn 4y ago
    I'm not. The rule to never trust user input will always be violated. The universe deems it so.
    • Ajedi32 4y ago
      I guess this now extends to "never trust AI input".
    • mathstuf 4y ago
      Seeing as it was trained on whatever it found on the Internet, it seems that "never trust user input" is far down the list of things to look at…
• nicholasjarnold 4y ago
  for the uninitiated - https://xkcd.com/327/ https://xkcd.com/327/ :)
• binarymax 4y ago
  Oh yes, little chatty tables, we call him.