9 ms·
Bard is much worse at puzzle solving than ChatGPT
- deleted 4y ago[deleted]
- SteveNuts 4y agoIt's so sad to me to see the downfall of google from the absolute coolest company on the planet to the one that's now trying to keep up.
- noflag 4y agos/Microsoft/Google/ http://www.paulgraham.com/microsoft.html http://www.paulgraham.com/microsoft.html
- eslaught 4y agoBut was Microsoft ever cool in the sense that Google was circa 2005? (Asking honestly, I'm too young to remember that period.)
- doctoboggan 4y agoNo, not really. They were seen as a serious business used by other serious businesses. IMO google pioneered the "cool tech company" with their 20% time, "don't be evil", unique campus life, etc.
- PopePompus 4y agoNo. I'm an old-timer, and there has not been one picosecond since the moment of its founding when Microsoft was cool.
- waihtis 4y agoNeed you even ask? https://www.youtube.com/watch?v=IY2j_GPIqRA https://www.youtube.com/watch?v=IY2j_GPIqRA
- MaximumYComb 4y agohttps://www.youtube.com/watch?v=Vhh_GeBPOhs&ab_channel=MrWueb007 https://www.youtube.com/watch?v=Vhh_GeBPOhs&ab_channel=MrWue... Really cool
- throw310822 4y agoYes, somehow. https://rarehistoricalphotos.com/windows-95-launch-day-1995/ https://rarehistoricalphotos.com/windows-95-launch-day-1995/ I remember people commenting "never seen such a thing before, for a computer OS!" "Many electronics stores held midnight launches for the product, with thousands of people waiting in line to be the first to get their hands on the operating system. The release was a tremendous success. Microsoft sold 7 million copies in the first five weeks, and Windows 95 was soon the most popular operating system on the market."
- prohobo 4y agoI know that at that time I used to be like "what kind of pleb uses anything other than Windows? Get off your high horse and use the OS that actually works", because well... Windows XP was pretty good, and games worked on it.
- leto_ii 4y agoFunny, cause Microsoft didn't die. They're doing great. More than that, they're the ones behind OpenAI.
- travisgriggs 4y agos/IBM/Microsoft/ History rhymes with itself.
- zedadex 4y ago> Microsoft Good joke. MS has always been in the 'incompetent evil' quadrant, Newcomers just keep inexplicably giving them the benefit of the doubt or assuming/insisting they've "changed".
- deleted 4y ago[deleted]
- mikewarot 4y agoIf I understand how Large Language models work, they don't actually know about spelling.... they are given tokens that represent words, and can only infer things from the context of those tokens across terabytes of data that they're given. Any rhyming done is an impressive result.
- ben0x539 4y agoI mean, English spelling certainly doesn't free you from the need to figure rhyming out from context.
- chewxy 4y agoWell the nature of byte pair encoding has a side effect of encoding sounds as well. I briefly explored it here: https://youtu.be/fTDQUxha9qU?t=480 https://youtu.be/fTDQUxha9qU?t=480
- masakreTech 4y ago[flagged]
- ramblerman 4y agoIt is amazing, but somewhat explicable as an emergent effect. Dont forget that the model has seen all the poems and raps on the internet. It built some latent space where certain words always cluster together in the context of poems, and in which location. In this case it really has the best database available to say, what next word would slot in nicely here - as that is precisely what it was trained to do. It is amazing, but somewhat explicable as an emergent effect. Dont forget that the model has seen all the poems and raps on the internet. It built some latent space where certain words always cluster together in the context of poems, and in which location. In this case it really has the best database available to say, what next word would slot in nicely here - as that is precisely how it was trained. I find it more amazing tbh that you can ask for a poem about something, and that it then sticks to the plot, makes references to the start etc than the actual rhyming.
- alfredohere 4y ago[dead]
- kodah 4y agoThat's a clever game to get it to play. Today I asked ChatGPT to give me 1000 Fibonacci numbers starting with the 2000th number and it crashed. Later I asked it the same prompt and it repeatedly gave me code to calculate the Fibonacci numbers in Python.
- Tiberium 4y ago> and it crashed I hope you understand that it's only because of inability of OpenAI's servers to keep up with demand or some issue in their backend code - language models themselves can't "crash" like normal programs on some kind of input, because they "just" generate new tokens.
- kodah 4y agoI know little more than what I've read about LLMs and other models. I just got it to do them again and it output them rather quickly: https://imgur.com/a/jfviCCe https://imgur.com/a/jfviCCe
- leto_ii 4y agoare the numbers correct?
- kodah 4y agoI spot checked an random assortment of them and so far yes, but it only takes one wrong one to set every subsequent number off.
- vineyardmike 4y ago> it only takes one wrong one to set every subsequent number off. Except its not actually doing that calculation, so one wrong one shouldn't truly affect the rest like "Real" math.
- mike_d 4y ago
- porphyra 4y agoHow do you navigate this blog to read the other articles? I couldn't find any way to read the one on gpt4 (clicking the underlined "wrote about" does nothing) and twofergoofer.com/blog goes to a 404.
- cldellow 4y agoI was curious to read it, too, so I googled to find the article. It's located at https://twofergoofer.com/blog/gpt-4 https://twofergoofer.com/blog/gpt-4
- cowllin 4y agoHah - we only made the blog over the weekend and don't have any nav or menu for now. But yep we link the prior article a few times in this article, that article goes into more detail!
- milemi 4y ago"Bard is much worse than ChatGPT at solving an obscure word game I invented" would have been a more honest title, but would probably generate less clicks for the author. Bard may still be much worse than ChatGPT at solving all kinds of puzzles, but the article is click bait for promoting the author's word game, not an actual investigation that warrants that conclusion.
- LesZedCB 4y agowhat's wrong with that? the use of novel puzzles is frankly awesome because there's a much lower chance of contamination from previous puzzles so we get a chance to see how much generalization they've achieved.
- milemi 4y agoI'm complaining about the title writing a check that the blog post can't cash.
- Skunkleton 4y agoGPT-4 says: A more accurate and balanced title might be: "Comparing Bard and ChatGPT in Puzzle Solving: An Examination within the Context of a Word Game"
- sebzim4500 4y agoSounds like GPT-4 will save us from clickbait
- cowllin 4y agoFair! But if I wrote Twofer Goofer in the title it would not resonate at at all. Alas, tradeoffs.
- deleted 4y ago[deleted]
- vineyardmike 4y ago
- hackpert 4y agoWow I had hoped for a more productive discussion than these 1-1 comparisons of Bard vs ChatGPT that I'm seeing everywhere. The model deployed with this version of Bard is clearly a smaller model than the biggest LaMDA/PaLM models Google has been working on for ages. Which, according to their publications, show unprecedented results on _proof writing_ of all things (see Minerva). While their strategic decisions may be questionable (or they're just trying to quantize the model for mass deployment without burning billions per month in compute costs), its almost silly to question Google's ability to build useful LLMs.
- random_cynic 4y ago> its almost silly to question Google's ability to build useful LLMs. Unless they release a model one can "use" and verify their claims it's literally silly to make this statement.
- amelius 4y agoThere are useful open source LLMs. Or are you questioning their ability to configure; make install?
- letitgo12345 4y agoEven the best Google models seem to be lagging for reasoning tasks vs OpenAI ones at the moment - see the graphs at https://github.com/suzgunmirac/BIG-Bench-Hard https://github.com/suzgunmirac/BIG-Bench-Hard
- xiphias2 4y agoIt seems like they don't want to be the best, just good and cheap enough that they don't lose users therefore ad revenue. They behave like Yahoo when Google took over.
- junofan 4y agoEveryone’s gonna have bigger models. Where are their useful models?
- seanhunter 4y ago
- fenomas 4y agoAm I missing something? Most of TFA is about Bard failing to answer with rhyming words, but in the only prompts shown the author doesn't actually ask for rhyming words. He just says the hint and the name of the puzzle. Is this not simply: "Bard is worse than ChatGPT at having seen the 'how-to-play' page for my side project during its training"?
- comex 4y agoClicking through to the link next to 'last week's text' and then to 'full rules', it looks like the author is starting the chat sessions with a full explanation that isn't included in the screenshots. (Also, the last screenshot shows the author explicitly asking about rhymes.)
- fenomas 4y agoAh thanks - for others here is the link, though TFA may not necessarily have used the same prompt: https://docs.google.com/document/d/1_eg_jiUE5y8e5zeiz5HCGDc2H2Jaw_u7n1nLzlKQAdM/edit# https://docs.google.com/document/d/1_eg_jiUE5y8e5zeiz5HCGDc2... Based on that it looks like the author asked all 25 test puzzles in one big prompt, which one supposes would favor larger models. To compare "puzzle solving" you'd think it would make more sense to ask one puzzle at a time?
- cowllin 4y agoI tried it both ways; with individual prompts and prompts in bulk. I ran both tests the same way. There's a tradeoff in writing a legible/interesting blog post and relating step-by-step the way the evaluation was ran! Appreciate you reading and the feedback :)
- jackblemming 4y agoHow is this possible? Google makes people do 8 rounds of leetcode. How could they be beaten? Nothing makes sense anymore.
- BoorishBears 4y agoPeople from Google have argued that's exactly why they're failing. Personally, as someone who worked at a company that was up over 500% during the pandemic, shipped absolutely nothing during that spike, and then deflated below their pre-pandemic pricing, I saw the foley in hiring smart people first hand. It's not enough to hire the smartest people, and in fact it can be a competitive disadvantage. The smartest people often want their piece of the product to reflect their ingenuity no matter how ancillary it is to the core mission. Unfortunately that often precludes the kind of agility that businesses need to stay competitive. OpenAI managed to poach Googlers by simply not having fiefdoms built by smart people™. I imagine if Google had built GPT-4, it wouldn't be having downtime today. Because it wouldn't be public. And it might never be public because it doesn't scale for Google scale yet, and the ethicists want their say, and we need to integrate it into Borg and the front end hasn't passed through enough layers of design and...
- natch 4y agoI think they just got lazy and entitled. Maybe ChatGPT will be the scare they need. It feels bad though; they almost don’t deserve the energy this fight will give them.
- NoZebra120vClip 4y agoI tried to play hangman with it, but it was on crack.
- masakreTech 4y agoBard is basically trash
- boffinism 4y ago> Twofer Goofer HQ's adherence to strict "perfect" rhyme can be tricky for those slant rhyme-inclined. And yet one puzzle they hammer Bard for failing is "Cactus Practice". What accent do you have to have for that to be a perfect rhyme?
- cowllin 4y agoFrom Chicago ... and with more than a thousand solves on that puzzle we've never received a single complaint about the rhyme on that one (plus the stats say users find it an extremely easy rhyme to solve)! Curious how you pronounce that one such that they don't rhyme?
- tetromino_ 4y agoAccording to the dictionary (and matching my own non-native pronunciation), cactus is /ˈkæktʌs/, practice is /ˈpɹæktɪs/ The terminal /tʌs/ is not quite the same thing as /tɪs/; since they are both unstressed, the difference can be hard to notice in fast speech, but becomes clear when enunciating.
- yorwba 4y agoDictionaries often neglect common mergers like the weak vowel merger https://en.wikipedia.org/wiki/Phonological_history_of_English_close_front_vowels#Weak_vowel_merger https://en.wikipedia.org/wiki/Phonological_history_of_Englis... where unstressed /ʌ/, /ə/ and /ɪ/ all end up being pronounced the same.
- runnerup 4y agoThere are differing definitions of “rhyme” in popular usage. See: https://youtu.be/_kQBVneC30o https://youtu.be/_kQBVneC30o A lot of “does this rhyme with that?” depends on the context in terms of how strict the rhyming must be or not.
- sethaurus 4y agoIn standard Australian English, this is a perfect rhyme.
- visarga 4y agoOfftopic - have you seen Phind? https://www.phind.com/ https://www.phind.com/ It is very fast and wins the search benchmarks here: https://twitter.com/vladquant/status/1638305110869807104 https://twitter.com/vladquant/status/1638305110869807104
- masakreTech 4y agoCan your source be not someone with an animated profile picture?
- visarga 4y agoSource's source: https://blog.kagi.com/kagi-ai-search#aitest https://blog.kagi.com/kagi-ai-search#aitest
- ralfd 4y agoIt is interesting how lackluster the reactions are about Bard, when it would have been jaw-gapping amazing just a year ago.