7 ms·
Just tried out Google Bard, now available in Europe, and I have to say, my first experience was rather disconcerting. I threw at it a relatively obscure questio
by frankohn 3y ago
Just tried out Google Bard, now available in Europe, and I have to say, my first experience was rather disconcerting. I threw at it a relatively obscure question about an event in the third book of A Song of Ice and Fire series, involving characters Arya Stark and Sandor Clegane.
To my surprise, Google Bard's response, while confident, was a bizarre mix of actual characters and situations from the book series, but the event it described was pure fabrication. It seemed as if it had hallucinated the entire scenario. Not exactly what I was expecting from such a sophisticated AI model.
As a point of comparison, I posed the exact same question to OpenAI's ChatGPT-4. It responded with a spot-on account of the event, complete with rich detail and an impressive level of accuracy. A stark contrast to Google Bard's confabulation.
In the light of this initial test, I can't say I'm ready to utilize Google Bard for anything serious just yet. The comparison with ChatGPT-4 is night and day. Despite this being just a single data point, the difference in quality and reliability was apparent. I'll be sticking with ChatGPT-4 for the time being.
- bsaul 3y agoAnyone here has ideas on what could make a difference between openai and bard on this issue ? Not an expert, so i really have no idea what it is that makes this kind of difference. training ? Architecture ? Rlhf ? Etc
- isoprophlex 3y agoDon't bother trying their other models available in GCP either. Their embedding models and their un-RLHFd generic GPT analogues are miles behind the competition. It's incredible how bad google have dropped the ball on this. We had some google people come in at $CORPO_DAYJOB the other day to sell their cloud offerings. The engineer had the gall to say "people ask me, why did google miss the boat on LLMs? I say people, we BUILT the boat". While referencing "attention is all you need". Hilarious coping strategy guys, but because you had some good academics on your payroll in 2017, that doesn't mean you're delivering right now... we'll talk when you actually have something better than 90% of the other LLM offerings out there.
- hgsgm 3y agoBuilding the boat and missing the boat are totally compatible. They guided others to a place they could not enter themselves.
- maxflow2 3y agoIt's also compatible with what I know about the business side of things. Google Brain/Research probably spent a lot of time and money in the neural network direction and still didn't have a clear way to productize it, so they cut them off just before they got there. Other companies were able to start where they left off since the research was public and make something with less time - making the investment in that direction look more attractive to higher ups.
- sebzim4500 3y agoYeah, I would say that "we built the boat" is simply a lie given none of the eight authors of the paper still work at Google.
- agnosticmantis 3y agoThat’s a strange take. Those researchers were given the resources, the right incentives, the right environment for that kind of research to happen, at Google, in a time when LLMs weren’t all the rage. “X Built a boat == the builders remain at X until they die” Is a strange definition of building something imo.
- lelanthran 3y agoI would say it's still correct. When a company provides an R&D lab, we still attribute the results of that research to the company as well as to the researchers. After all, if those individuals were not hired that position would still be filled with someone similar, but it's hard to argue that if the lab didn't bring those researchers together, they may have individually gotten the same result. Honestly, it's not as easy as one may think to build a research lab.
- kernal 3y ago
- ChatGTP 3y agoQuestion, do you think this has to do with copyrighted information being in the models ? I know you might not care but I wonder if Google is operating of accounts from the internet while OpenAI has actually ingested the books ?
- frankohn 3y agoI don't know but I think google bard had access to the book's content as well because it mentioned correctly some characters and some places I didn't mention in my question.
- amf12 3y agoThat's exactly what LLMs are good at. Cross-referencing information from across the training data. The response could contain information that you did not specifically refer to but is relevant and/or related. It doesn't even imply that the LLMs had access to the book's content.
- buildbot 3y agoNo. A blog post on the internet is also copyrighted! Google has probably the largest collection of digitized books as well. They have no reason not to. ML networks have been trained on copyrighted data since before 2012.
- munificent 3y ago> They have no reason not to. Ethics?
- buildbot 3y agoPersonally, I have no ethical issue with it. This response is copyrighted, all rights reserved. But you can still read and gather information from it. Your computer still stores a copy of it when you reload the page. Theft? Infringement? I think not. Is taking the character count of a book copyright infringement? Why is the math behind an LLM different?
- minsc_and_boo 3y agoWhat was the prompt?
- frankohn 3y agoIn the book the The game of Thrones what happens at Saltpans with Arya and Sandor Clegane ? Here the link with the answer of ChatGPT: https://chat.openai.com/share/7c46a3bd-efa8-49e9-8018-554e1ebf18f8 https://chat.openai.com/share/7c46a3bd-efa8-49e9-8018-554e1e... and also a follow up question.
- stevepike 3y agoIs this actually correct? https://awoiaf.westeros.org/index.php/Saltpans https://awoiaf.westeros.org/index.php/Saltpans makes me think ChatGPT is referencing the wrong book.
- alanfranz 3y ago> As a point of comparison, I posed the exact same question to OpenAI's ChatGPT-4. It responded with a spot-on account of the event, complete with rich detail and an impressive level of accuracy. Which, most likely, implies that chatgpt was trained and retains actual book content rather than random people talking about it on the internet. Which model is more likely to be in violation of some copyright?
- replwoacause 3y agoThe ChatGPT one, but that isn’t my problem. My concern is whether it gives me good information. The issue of copyright is an unfortunate issue for someone else to litigate.
- olddustytrail 3y agoGood point. If I ever want a random summary of an event in a fantasy book I've already read, I'll be sure to stick to ChatGPT. On a completely unrelated point, why are people's tests of LLMs clearly designed to make me think that people are dumber? Are the LLMs suggesting these questions as part of their plot?
- mrtranscendence 3y agoGreat. And if I want your opinion on how smart it is to be a bit silly with LLMs when I’m not even attempting to test them systematically, I’ll be sure to ask you.
- olddustytrail 3y agoWhat's it got to do with you?