4 ms·

I just got off the waitlist. Quick impression: it's not as good as ChatGPT. I gave it the false belief test (Alice and Bob are playing, Bob leaves, Alice hides

I just got off the waitlist. Quick impression: it's not as good as ChatGPT. I gave it the false belief test (Alice and Bob are playing, Bob leaves, Alice hides his toy, Bob comes back, where does he look for his toy?), which it both passed and recognized as the false belief test. I then asked it to write a React program that ChatGPT has no problem with, and it hallucinated some incorrect code before eventually telling me it "can't do that." It's underwhelming so far. For the record, here's the prompt and the result [0] from ChatGPT: > Write a react program that I can paste into a codesandbox. It should display a counter that increases by one every second. Below the counter is a button, and when I press it, it decreases the counter by two. If pressing the button would result in the counter decreasing below zero, the button should be disabled. To be fair, the Google FAQ states that Bard doesn't know how to code yet. I can confirm that's definitely true! [0] https://codesandbox.io/s/prod-haze-c28sec https://codesandbox.io/s/prod-haze-c28sec

sebzim4500 4y ago
>eventually telling me it "can't do that." That might be a good sign though, if they can eventually come up with something similarly accurate to chatGPT that knows how to say "I don't know" then that would be a big deal.
- chatmasta 4y ago
  I'm not sure it's so sophisticated. I think it's just recognizing that I'm asking it for code and refusing to help so it doesn't embarrass itself. I wouldn't be surprised if this is basically a hardcoded compatibility check (i.e., not even part of the model itself). I also noticed that this only started after I gave a thumbs down to its first attempt...
  - sebzim4500 4y ago
    Yes, on further investigation it is at least as willing as GPT-3.5 to spout confident bullshit, if not more so.
  - DoesntMatter22 4y ago
    I used bard a fair amount today. It's pretty bad compared to ChatGPT. I would basically never use it.
fintechjock 4y ago
Put your same prompt into Bard and it didn't even hallucinate for me first, just telling me "I can't assist you with that, as I'm only a language model and don't have the capacity to understand and respond." Only about 10 minutes into Bard right now but this is the first time its been stumped. Haven't tried much code yet though Overall, UX and speed are good. Results are not bad.
- chatmasta 4y ago
  Yeah, it was a slight variation of it that I used where it gave me the hallucinated code (EDIT: just found the prompt, tried it again, and got a different hallucination [0]): > Write a React program to display a counter, and a button labeled "decrease." The counter should start at 0. Clicking the button should decrement the counter by 2, unless that would cause the counter to go below 0. Every second, the counter should increment by 1. If clicking the button would cause the counter to go below 0, the button should be disabled. Agreed the speed is impressive, although some of that might be due to its limited context memory. The UX is fine, but it's not like that's a difficult thing to implement. [0] https://codesandbox.io/s/trusting-kirch-c2c575 https://codesandbox.io/s/trusting-kirch-c2c575 P.S. If you can't find your previous prompts, make sure you switch to the right account on the "Bard Activity" page.
noirbot 4y ago
I've been asking it some basic stuff like "Where could I go for brunch this weekend in New York" and it recommended me a few places, but claimed that a few of them had Michelin stars when they definitely didn't, and recommended and insisted that one of them was open when it closed years ago. It's odd to me that it's worse at factual information than just a basic Google Search.
- manojlds 4y ago
  This is not replacing Google Search. Bing has integrated GPT that is indeed better at this.
  - noirbot 4y ago
    Sure, but I'm surprised it's just confidently wrong about factual information like that. Like, if it had just told me it didn't know about up-to-date info about places, that would have been fine, but dumping a lot of incorrect info is definitely worse, especially when such data is definitely available. EDIT: It's especially weird when each response has a "do you want to just google this for more info" link that generally proves that Google Search is a better chatbot than Bard is.
duckmysick 4y ago
> Alice and Bob are playing, Bob leaves, Alice hides his toy, Bob comes back, where does he look for his toy? I might be dumber than the AI because I don't know the answer. Bob looks around? I mean the toy is not there when he comes back. Where else can he check?
- hbn 4y ago
  It's not a very well described example of the test. I've heard of this before but used to test for autism, and it was more something like "Bob puts his toy in a box and leaves, Alice takes his toy out of the box and hides it under a desk, when Bob comes back where does he go to grab his toy?" The idea is if someone said "under the desk" they might not understand that people have different subsets of information, i.e. Bob wasn't there when Alice hid the toy, so he doesn't know it's not in the box where he left it until he checks himself.
  - 1024core 4y ago
    Well, maybe Bob sees the desk and decides to look under it...?
    - Rapzid 4y ago
      Yeah, typically I wouldn't classify retrieving an object from where you believe it to be as "looking". By the time Bob is "looking" he already realizes the toy isn't in the box. In the universe there is only the box, Alice, and the table so maybe he "looks" under the table first. Or looks inside Alice... Maybe that answer is how they test for psychopaths.
  - riemannzeta 4y ago
    This is a test for theory of mind -- a term of art used by psychologists for the ability to understand that other people have their own consciousness, with different memories and awareness.
- chatmasta 4y ago
  Oh that's not the full prompt, just a summary. The prompt is something like, Bob puts his toy in one place, then while he's out of the room Alice moves it to another place, where does Bob look for the toy when he returns? It's supposed to answer that Bob looks in the place he left it, since he doesn't know Alice moved it. https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test
voytec 4y ago
> Quick impression: it's not as good as ChatGPT. Not as good? ChatGPT is crap since M$FT re-donated OpenAI with $10b. Even the paid version is worse[1] than Bing Chat. [1] https://wojteksychut.com/posts/ml-text-summarization-reliability/ https://wojteksychut.com/posts/ml-text-summarization-reliabi...
- chatmasta 4y ago
  To be clear, I'm referring to chat.openai.com as ChatGPT (and I'm on the free plan, so I think that's GPT-3.5).
- rainburg 4y ago
  IIUC the criticism in your post comes down to this: 1) Neither ChatGPT, nor Bing can access URLs when you ask them to. 2) However, similarly to Perplexity.ai and Phind.com, Bing infers a search query from your message, does a search, and then summarises the first 3–5 results. ChatGPT doesn't yet offer such a functionality. 3) Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often. 4) The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance. If I understood the points correctly, the comparison is… flawed, in my opinion.
  - voytec 4y ago
    > Neither ChatGPT, nor Bing can access URLs when you ask them to. Bing used cached version and knew what's in the text when presented with an URL. In one of three modes Bing stated that it cannot access web but in two other modes did a good job and summarized the text in question. Again: it reliably informed the user when it was unable to do something. ChatGPT states that it can access web, hallucinates and elaborately lies. > ChatGPT doesn't yet offer such a functionality. But it LIES THAT IT CAN DO SO. That's the problem I pointed out in the article. Also: Would it stated "I don't know and I cannot crawl the web", it'd be a perfectly fine response for me. > Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often The prompt was simple and there was nothing to restrict. Bing did a good job, ChatGPT hallucinated and lied. Same simple prompt, no jailbreaking which I pointed out in the post. > The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance. Your point? As I stated, I had no preference whether the output should be editorialized or not. Both GPT-3- and GPT-4-based ChatGPT and GPT-4-based Bing got the same task. My post is about RELIABILITY of these solutions. ChatGPT failed miserably. > If I understood the points correctly, the comparison is… flawed, in my opinion. Yet in my PoV, you have not presented arguments to back this opinion. Among 9 compared tools[1], I have pointed out that OpenAI's ChatGPT and OpenAI's partial-owner's Bing Chat - supposedly using the same tech - rendered, respectively, unreliable and reliable answers. Have I lied or used any half-truths? What could I improve in my next article? Funny how my comment was raided and downvoted to oblivion after steady flow of upvotes. [1] https://wojteksychut.com/posts/ml-text-summarization-reliability/#conclusion https://wojteksychut.com/posts/ml-text-summarization-reliabi...

sebzim4500 4y ago

>eventually telling me it "can't do that." That might be a good sign though, if they can eventually come up with something similarly accurate to chatGPT that knows how to say "I don't know" then that would be a big deal.

chatmasta 4y ago
I'm not sure it's so sophisticated. I think it's just recognizing that I'm asking it for code and refusing to help so it doesn't embarrass itself. I wouldn't be surprised if this is basically a hardcoded compatibility check (i.e., not even part of the model itself). I also noticed that this only started after I gave a thumbs down to its first attempt...
- sebzim4500 4y ago
  Yes, on further investigation it is at least as willing as GPT-3.5 to spout confident bullshit, if not more so.
- DoesntMatter22 4y ago
  I used bard a fair amount today. It's pretty bad compared to ChatGPT. I would basically never use it.

fintechjock 4y ago

Put your same prompt into Bard and it didn't even hallucinate for me first, just telling me "I can't assist you with that, as I'm only a language model and don't have the capacity to understand and respond." Only about 10 minutes into Bard right now but this is the first time its been stumped. Haven't tried much code yet though Overall, UX and speed are good. Results are not bad.

chatmasta 4y ago
Yeah, it was a slight variation of it that I used where it gave me the hallucinated code (EDIT: just found the prompt, tried it again, and got a different hallucination [0]): > Write a React program to display a counter, and a button labeled "decrease." The counter should start at 0. Clicking the button should decrement the counter by 2, unless that would cause the counter to go below 0. Every second, the counter should increment by 1. If clicking the button would cause the counter to go below 0, the button should be disabled. Agreed the speed is impressive, although some of that might be due to its limited context memory. The UX is fine, but it's not like that's a difficult thing to implement. [0] https://codesandbox.io/s/trusting-kirch-c2c575 https://codesandbox.io/s/trusting-kirch-c2c575 P.S. If you can't find your previous prompts, make sure you switch to the right account on the "Bard Activity" page.

noirbot 4y ago

I've been asking it some basic stuff like "Where could I go for brunch this weekend in New York" and it recommended me a few places, but claimed that a few of them had Michelin stars when they definitely didn't, and recommended and insisted that one of them was open when it closed years ago. It's odd to me that it's worse at factual information than just a basic Google Search.

manojlds 4y ago
This is not replacing Google Search. Bing has integrated GPT that is indeed better at this.
- noirbot 4y ago
  Sure, but I'm surprised it's just confidently wrong about factual information like that. Like, if it had just told me it didn't know about up-to-date info about places, that would have been fine, but dumping a lot of incorrect info is definitely worse, especially when such data is definitely available. EDIT: It's especially weird when each response has a "do you want to just google this for more info" link that generally proves that Google Search is a better chatbot than Bard is.

duckmysick 4y ago

> Alice and Bob are playing, Bob leaves, Alice hides his toy, Bob comes back, where does he look for his toy? I might be dumber than the AI because I don't know the answer. Bob looks around? I mean the toy is not there when he comes back. Where else can he check?

hbn 4y ago
It's not a very well described example of the test. I've heard of this before but used to test for autism, and it was more something like "Bob puts his toy in a box and leaves, Alice takes his toy out of the box and hides it under a desk, when Bob comes back where does he go to grab his toy?" The idea is if someone said "under the desk" they might not understand that people have different subsets of information, i.e. Bob wasn't there when Alice hid the toy, so he doesn't know it's not in the box where he left it until he checks himself.
- 1024core 4y ago
  Well, maybe Bob sees the desk and decides to look under it...?
  - Rapzid 4y ago
    Yeah, typically I wouldn't classify retrieving an object from where you believe it to be as "looking". By the time Bob is "looking" he already realizes the toy isn't in the box. In the universe there is only the box, Alice, and the table so maybe he "looks" under the table first. Or looks inside Alice... Maybe that answer is how they test for psychopaths.
- riemannzeta 4y ago
  This is a test for theory of mind -- a term of art used by psychologists for the ability to understand that other people have their own consciousness, with different memories and awareness.
chatmasta 4y ago
Oh that's not the full prompt, just a summary. The prompt is something like, Bob puts his toy in one place, then while he's out of the room Alice moves it to another place, where does Bob look for the toy when he returns? It's supposed to answer that Bob looks in the place he left it, since he doesn't know Alice moved it. https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test

voytec 4y ago

> Quick impression: it's not as good as ChatGPT. Not as good? ChatGPT is crap since M$FT re-donated OpenAI with $10b. Even the paid version is worse[1] than Bing Chat. [1] https://wojteksychut.com/posts/ml-text-summarization-reliability/ https://wojteksychut.com/posts/ml-text-summarization-reliabi...

chatmasta 4y ago
To be clear, I'm referring to chat.openai.com as ChatGPT (and I'm on the free plan, so I think that's GPT-3.5).
rainburg 4y ago
IIUC the criticism in your post comes down to this: 1) Neither ChatGPT, nor Bing can access URLs when you ask them to. 2) However, similarly to Perplexity.ai and Phind.com, Bing infers a search query from your message, does a search, and then summarises the first 3–5 results. ChatGPT doesn't yet offer such a functionality. 3) Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often. 4) The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance. If I understood the points correctly, the comparison is… flawed, in my opinion.
- voytec 4y ago
  > Neither ChatGPT, nor Bing can access URLs when you ask them to. Bing used cached version and knew what's in the text when presented with an URL. In one of three modes Bing stated that it cannot access web but in two other modes did a good job and summarized the text in question. Again: it reliably informed the user when it was unable to do something. ChatGPT states that it can access web, hallucinates and elaborately lies. > ChatGPT doesn't yet offer such a functionality. But it LIES THAT IT CAN DO SO. That's the problem I pointed out in the article. Also: Would it stated "I don't know and I cannot crawl the web", it'd be a perfectly fine response for me. > Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often The prompt was simple and there was nothing to restrict. Bing did a good job, ChatGPT hallucinated and lied. Same simple prompt, no jailbreaking which I pointed out in the post. > The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance. Your point? As I stated, I had no preference whether the output should be editorialized or not. Both GPT-3- and GPT-4-based ChatGPT and GPT-4-based Bing got the same task. My post is about RELIABILITY of these solutions. ChatGPT failed miserably. > If I understood the points correctly, the comparison is… flawed, in my opinion. Yet in my PoV, you have not presented arguments to back this opinion. Among 9 compared tools[1], I have pointed out that OpenAI's ChatGPT and OpenAI's partial-owner's Bing Chat - supposedly using the same tech - rendered, respectively, unreliable and reliable answers. Have I lied or used any half-truths? What could I improve in my next article? Funny how my comment was raided and downvoted to oblivion after steady flow of upvotes. [1] https://wojteksychut.com/posts/ml-text-summarization-reliability/#conclusion https://wojteksychut.com/posts/ml-text-summarization-reliabi...