4 ms·

I'm definitely perplexed but not in a good way. It's fast but... I asked it what it was and first it told me it was Google Lens. Then it told me some random s

I'm definitely perplexed but not in a good way. It's fast but... I asked it what it was and first it told me it was Google Lens. Then it told me some random stuff about android. I asked "But what are you though?" and got this: "A black hole is a region in space with extremely strong gravity, so powerful that not even light can escape it. They form from the remnants of large stars that die in supernova explosions. Key features of a black hole include the event horizon, which is the boundary in spacetime where nothing can escape, and the singularity, a point at the center of a black hole where spacetime curvature becomes infinite. Most black holes are 10 to 24 times as massive as the Sun. Although they cannot be directly observed, scientists can study black holes by detecting their effects on nearby matter."

kylebenzle 3y ago
Just tried and got the same odd response. Maybe "what is this" or is a common phrase searched for that leads to Google Lens? No matter what, Perplexity is now the worst of the worst. They were early with the ability to upload documents but the utter failure of Perplexity to be useful is proving what I have been saying for a year now, (1) LLMs are not "AI" any more than a spell checker is and (2) LLMs are not really useful for anything other than a VERY good spell checker.
- sdenton4 3y ago
  I'm not sure why 'perplexity is bad' means 'all LLMs are nearly useless.' Do we use Warwick Davis as the reference height for humans?
  - Exoristos 3y ago
    Procrustes salivates.
- polynomial 3y ago
  While I believe the impact of LLMs is often overstated, I'd like to hear what you really mean that they are not AI any more than spell checker. Arguably an LLM is significantly closer to AI than a spell checker.
  - kylebenzle 3y ago
    I don't see LLMs being close to "real" AI at all and Sam Altman also has said he does not see LLMs ever leading to AGI but also says he could be wrong. As to why an LLM is not AI? People are wrongfully "afraid" that an LLM is somehow make "bad" information too easily available to people. Like how to rob a bank or make a bomb. The confusion is in thinking that an LLM can output something that it had not seen in training and supposedly all the training data is available elsewhere. Now, "data leaking" is a serious issue with LLMs but that has nothing to do with "AI". An LLM is a good auto-complete, which is a good spell checker, which is a good search algorithm. I don't see how we ever get from context search and retrieval to "AI".
- andybak 3y ago
  I thought I was finding LLMs useful but you seem to be so sure of yourself, that I surely must have been mistaken.
- todd3834 3y ago
  While I understand what you are trying to say with your analogy I think it shows a misunderstanding of how they work. If you meant a very good autocomplete then yes that's exactly what they are. However, predicting the next word in an autocomplete can be naive or rather involved. > not really useful for anything other than a VERY good spell checker. Try GitHub code pilot. Try asking it to summarize something you know nothing about and then tell it that you need the explanation to be more simple several times. How can you compare this to a spell checker?
  - Nullabillity 3y ago
    > If you meant a very good autocomplete then yes that's exactly what they are. However, predicting the next word in an autocomplete can be naive or rather involved. People keep trying to apply them as autocompleters... that doesn't mean they're good at it.
- seanhunter 3y ago
  Serious question: Why do you think asking it what it is constitutes a good test of its capabilities? Do you think (for example) someone asking you what you are would give them enough ability to judge your abilities? Yes you would get the answer right and the models you are choosing are getting the answer wrong, but you can't extrapolate from that to anything else at all about what this model can or can't do.
  - brookst 3y ago
    Most people, me included, would probably not give a sufficiently sophisticated answer to “what are you” to satisfy a philosopher who’d spent years examining the question and the body of literature attempting to answer it. It’s a little silly to test a small LLM with a question that at least requires knowledge of its own construction that was not included in its training set, and which really requires existential introspection.
raylad 3y ago
Which model did you pick? It defaults to 7B which wouldn't be expected to be the brightest of the bunch. If you pick their 70B model it says: "I am Perplexity".
- stavros 3y ago
  Mistral 7B is very good, definitely way better than such confusion.
  - furyofantares 3y ago
    The default here is pplx-7b-online which I'm thinking is not instruct-tuned. There's also pplx-7b-chat which doesn't appear to exhibit this confusion and I think is instruct-tuned. Very strange default for them to choose imo.
    - stavros 3y ago
      Ah, yes, just today a friend was having lots of trouble with Mixtral returning terrible results, until he got Mixtral-instruct. Very interesting how much better the UX of instruct models is.
      - cosmojg 3y ago
        I think it's a matter of setting expectations appropriately. LLMs aren't chatbots by default, they're text prediction engines. Depending on the use case, the instruct-tuned models can be quite a bit more difficult to use.
        stavros 3y ago
        For the average person, the text completion mode is very unintuitive. Even my friend, who's a very experienced developer, had issues with the non-instruct model, even after I told him he needs to structure his queries as completions.
- quickthrower2 3y ago
Havoc 3y ago
>what it was >But what are you though? LLMs don't do well with that little context and words that are essentially "empty". They occur in all sorts of contexts - like black hole articles.
robbiep 3y ago
It gave me a black hole answer as well when I asked it what its sources were for a statement it output
rajeevk 3y ago
I tried asking a very specific objective question: what is the speed of light? Although it answered, but it also spitted lot of unnecessary stuff. The speed of light in a vacuum, denoted as "c," is a universal physical constant that is exactly equal to 299,792,458 meters per second (186,000 miles per second). This constant is the upper limit for the speed at which conventional matter or energy, and thus any signal, can travel. According to the special theory of relativity, the speed of light is constant and does not change with the motion of the observer. The speed of light has significant implications in various fields, including physics, computing, and communication. In physics, it serves as the single limiting velocity in the universe, being an upper bound to the propagation speed of signals and the speeds of all material particles. In computing, the speed of light fixes the ultimate minimum communication delay. In time of flight measurements, the speed of light can be used to measure large distances. The speed of light was first measured to high precision in 1879 by the Michelson-Morley Experiment. Since then, it has been used to define international standard measurements like the meter, mile, foot, and inch. It is more like a search engine. In fact Google search does much better than this. It gives very precise answer.
- Kranar 3y ago
  Your question wasn't specific though. A specific question would be how fast does light travel in a vacuum. When I asked it that specific question it replied in two sentences.
lqet 3y ago
The 70b-chat model is definitely better: > Can you write a poem in biblical style praising an aubergine? In the beginning, there was the aubergine, A fruit of the vine, a treasure so fine, In the realm of the earth, it stood tall and proud, A testament to the heavens, a voice so loud. Bearing the mark of the Creator's hand, A symbol of life, a promise so grand, The aubergine held the secrets of the soil, A repository of wisdom, a gem so precious and royal. Her skin, a tapestry of midnight hue, A dance of shadows, a harmony so true, In her heart, a gemstone of emerald green, A symphony of flavors, a divine sovereign. etc, it went on for 10 more verses.

kylebenzle 3y ago

Just tried and got the same odd response. Maybe "what is this" or is a common phrase searched for that leads to Google Lens? No matter what, Perplexity is now the worst of the worst. They were early with the ability to upload documents but the utter failure of Perplexity to be useful is proving what I have been saying for a year now, (1) LLMs are not "AI" any more than a spell checker is and (2) LLMs are not really useful for anything other than a VERY good spell checker.

sdenton4 3y ago
I'm not sure why 'perplexity is bad' means 'all LLMs are nearly useless.' Do we use Warwick Davis as the reference height for humans?
- Exoristos 3y ago
  Procrustes salivates.
polynomial 3y ago
While I believe the impact of LLMs is often overstated, I'd like to hear what you really mean that they are not AI any more than spell checker. Arguably an LLM is significantly closer to AI than a spell checker.
- kylebenzle 3y ago
  I don't see LLMs being close to "real" AI at all and Sam Altman also has said he does not see LLMs ever leading to AGI but also says he could be wrong. As to why an LLM is not AI? People are wrongfully "afraid" that an LLM is somehow make "bad" information too easily available to people. Like how to rob a bank or make a bomb. The confusion is in thinking that an LLM can output something that it had not seen in training and supposedly all the training data is available elsewhere. Now, "data leaking" is a serious issue with LLMs but that has nothing to do with "AI". An LLM is a good auto-complete, which is a good spell checker, which is a good search algorithm. I don't see how we ever get from context search and retrieval to "AI".
andybak 3y ago
I thought I was finding LLMs useful but you seem to be so sure of yourself, that I surely must have been mistaken.
todd3834 3y ago
While I understand what you are trying to say with your analogy I think it shows a misunderstanding of how they work. If you meant a very good autocomplete then yes that's exactly what they are. However, predicting the next word in an autocomplete can be naive or rather involved. > not really useful for anything other than a VERY good spell checker. Try GitHub code pilot. Try asking it to summarize something you know nothing about and then tell it that you need the explanation to be more simple several times. How can you compare this to a spell checker?
- Nullabillity 3y ago
  > If you meant a very good autocomplete then yes that's exactly what they are. However, predicting the next word in an autocomplete can be naive or rather involved. People keep trying to apply them as autocompleters... that doesn't mean they're good at it.
seanhunter 3y ago
Serious question: Why do you think asking it what it is constitutes a good test of its capabilities? Do you think (for example) someone asking you what you are would give them enough ability to judge your abilities? Yes you would get the answer right and the models you are choosing are getting the answer wrong, but you can't extrapolate from that to anything else at all about what this model can or can't do.
- brookst 3y ago
  Most people, me included, would probably not give a sufficiently sophisticated answer to “what are you” to satisfy a philosopher who’d spent years examining the question and the body of literature attempting to answer it. It’s a little silly to test a small LLM with a question that at least requires knowledge of its own construction that was not included in its training set, and which really requires existential introspection.

raylad 3y ago

Which model did you pick? It defaults to 7B which wouldn't be expected to be the brightest of the bunch. If you pick their 70B model it says: "I am Perplexity".

stavros 3y ago
Mistral 7B is very good, definitely way better than such confusion.
- furyofantares 3y ago
  The default here is pplx-7b-online which I'm thinking is not instruct-tuned. There's also pplx-7b-chat which doesn't appear to exhibit this confusion and I think is instruct-tuned. Very strange default for them to choose imo.
  - stavros 3y ago
    Ah, yes, just today a friend was having lots of trouble with Mixtral returning terrible results, until he got Mixtral-instruct. Very interesting how much better the UX of instruct models is.
    - cosmojg 3y ago
      I think it's a matter of setting expectations appropriately. LLMs aren't chatbots by default, they're text prediction engines. Depending on the use case, the instruct-tuned models can be quite a bit more difficult to use.
      - stavros 3y ago
        For the average person, the text completion mode is very unintuitive. Even my friend, who's a very experienced developer, had issues with the non-instruct model, even after I told him he needs to structure his queries as completions.
quickthrower2 3y ago

Havoc 3y ago

>what it was >But what are you though? LLMs don't do well with that little context and words that are essentially "empty". They occur in all sorts of contexts - like black hole articles.

robbiep 3y ago

It gave me a black hole answer as well when I asked it what its sources were for a statement it output

rajeevk 3y ago

I tried asking a very specific objective question: what is the speed of light? Although it answered, but it also spitted lot of unnecessary stuff. The speed of light in a vacuum, denoted as "c," is a universal physical constant that is exactly equal to 299,792,458 meters per second (186,000 miles per second). This constant is the upper limit for the speed at which conventional matter or energy, and thus any signal, can travel. According to the special theory of relativity, the speed of light is constant and does not change with the motion of the observer. The speed of light has significant implications in various fields, including physics, computing, and communication. In physics, it serves as the single limiting velocity in the universe, being an upper bound to the propagation speed of signals and the speeds of all material particles. In computing, the speed of light fixes the ultimate minimum communication delay. In time of flight measurements, the speed of light can be used to measure large distances. The speed of light was first measured to high precision in 1879 by the Michelson-Morley Experiment. Since then, it has been used to define international standard measurements like the meter, mile, foot, and inch. It is more like a search engine. In fact Google search does much better than this. It gives very precise answer.

Kranar 3y ago
Your question wasn't specific though. A specific question would be how fast does light travel in a vacuum. When I asked it that specific question it replied in two sentences.

lqet 3y ago

The 70b-chat model is definitely better: > Can you write a poem in biblical style praising an aubergine? In the beginning, there was the aubergine, A fruit of the vine, a treasure so fine, In the realm of the earth, it stood tall and proud, A testament to the heavens, a voice so loud. Bearing the mark of the Creator's hand, A symbol of life, a promise so grand, The aubergine held the secrets of the soil, A repository of wisdom, a gem so precious and royal. Her skin, a tapestry of midnight hue, A dance of shadows, a harmony so true, In her heart, a gemstone of emerald green, A symphony of flavors, a divine sovereign. etc, it went on for 10 more verses.