3 ms·

I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was b

I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was born in San Francisco and it apologized and then said his father was a missionary traveling in America, which was also wrong. Bruce’s father was a famous Cantonese Opera singer and actor. This model had zero information right, while being fast in responding. Unacceptable.

selcuka 2mo ago
It gave the correct answers to both questions for me: > Bruce Lee was born in San Francisco, California, USA on November 27, 1940. > Bruce Lee's father was a Chinese opera singer That being said, this is not a good test. It is a language model (a very small one), not an encyclopedia. ChatJimmy interface is just a tech demo. Without tool calling functionality we can't expect it to be factually correct.
- logicallee 2mo ago
  if it's baked into silicon how can you two get different answers?
  - v9v 2mo ago
    It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
    - logicallee 2mo ago
      Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
      - Tuna-Fish 2mo ago
        You can make any LLM deterministic by dropping the temperature hyperparameter to zero. This will generally make them suck, though, a little bit of randomness is necessary for proper function.
        fwip 2mo ago
        You can also use a fixed seed for your prng. A hash of the input text (up to the current turn) should do.
    - fph 2mo ago
      But since it's so fast you can just ask it 100 times where Bruce Lee was born, and statistically you'll get the correct answer. We could call it "mixture of idiots". /s
TeMPOraL 2mo ago
That's not what speed is useful for. I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}. Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.
- logicallee 2mo ago
  >That's not what speed is useful for. >I just pasted your comment and its whole inheritance chain to it, Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt: >did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread: followed by pasting the thread[1] And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible. It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct! So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless. [1] prompt: https://ibb.co/CKVmRvtd https://ibb.co/CKVmRvtd [2] result: https://ibb.co/BKdRKmyD https://ibb.co/BKdRKmyD
tliltocatl 2mo ago
Using LLMs for information retrieval is the most stupid thing one can do. Especially when old methods work much better.

selcuka 2mo ago

It gave the correct answers to both questions for me: > Bruce Lee was born in San Francisco, California, USA on November 27, 1940. > Bruce Lee's father was a Chinese opera singer That being said, this is not a good test. It is a language model (a very small one), not an encyclopedia. ChatJimmy interface is just a tech demo. Without tool calling functionality we can't expect it to be factually correct.

logicallee 2mo ago
if it's baked into silicon how can you two get different answers?
- v9v 2mo ago
  It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
  - logicallee 2mo ago
    Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
    - Tuna-Fish 2mo ago
      You can make any LLM deterministic by dropping the temperature hyperparameter to zero. This will generally make them suck, though, a little bit of randomness is necessary for proper function.
      - fwip 2mo ago
        You can also use a fixed seed for your prng. A hash of the input text (up to the current turn) should do.
  - fph 2mo ago
    But since it's so fast you can just ask it 100 times where Bruce Lee was born, and statistically you'll get the correct answer. We could call it "mixture of idiots". /s

TeMPOraL 2mo ago

That's not what speed is useful for. I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}. Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.

logicallee 2mo ago
>That's not what speed is useful for. >I just pasted your comment and its whole inheritance chain to it, Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt: >did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread: followed by pasting the thread[1] And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible. It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct! So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless. [1] prompt: https://ibb.co/CKVmRvtd https://ibb.co/CKVmRvtd [2] result: https://ibb.co/BKdRKmyD https://ibb.co/BKdRKmyD

tliltocatl 2mo ago

Using LLMs for information retrieval is the most stupid thing one can do. Especially when old methods work much better.