10 ms·
Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the h
by jug 3y ago
Still fails my hippo test!
> Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals.
But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.
- Jaygles 3y agoGoogle says that swimming is "the sport or activity of propelling oneself through water using the limbs". It doesn't constrain the propulsion to only be between the limbs and the water. Seems like pushing against the ground to move through the water fits the definition
- jug 3y agoGeneral consensus if you ask about it is that it's not considered swimming though. I think you'd be looking for locomotion through water without frequent touch downs. This is an example: https://www.nationalgeographic.com/animals/mammals/facts/hippopotamus https://www.nationalgeographic.com/animals/mammals/facts/hip... GPT-4 had a much better answer last I checked. Edit: Updated with another non-paywall link.
- canttestthis 3y agoAfter having reviewed the relevant material I think it's fair to say that hippos swim, per our conventional understanding of what swimming is.
- simbolit 3y agoAnd I think it's fair to say they don't, per conventional understanding. Good for both of us.
- TaylorAlexander 3y agoDictionary definitions and language use in practice do not always overlap 100%. The true meaning of a word comes from its usage, not the dictionary.
- deleted 3y ago[deleted]
- 8note 3y agoThe impact of this, I think is that the LLM is right, hippos do swim, because that's how people commonly describe what hippos do, such that it gets into the training set
- TaylorAlexander 3y agoI don't think so. I think what is happening is that most people are not familiar with hippos because probably somewhere between zero and one people in this thread have ever seen them in their natural habitat. The average person might think hippos swim because they do not understand that hippos do not float. If you were to put a hippo in a very deep pool they would simply drown, because they cannot float nor swim. I think if you clarified this, most people would not continue to assert that they are swimming. Specifically I found these citations very helpful: https://news.ycombinator.com/item?id=36777376 https://news.ycombinator.com/item?id=36777376 So I think what we can say is that the average person would be wrong about hippos, because the hippo is not actually swimming but people do not realize that. We expect LLMs to behave more like an expert, so the standard is even higher and we should be more willing to say it is wrong. Although the meaning of a word is defined by its usage, there are also common misconceptions that people have. It is not the case that every usage of a word is correct. Instead you would have to ask a variety of people what swimming is, and then describe to them what a hippo does (it does not float, it walks on the river bed), and then consider whether that fits with their conception of swimming. I think what is happening here is that lots of people thought hippos swim, they have been corrected, and now they are feeling a bit defensive and trying to save face rather than say "huh that is interesting I did not know that hippos don't swim".
- jeroen 3y agoUsing that interpretation of the definition, you could argue that walking through knee-deep water is swimming.
- earthboundkid 3y agoMy two year old can walk in water but cannot swim.
- dTal 3y agoThat's a very odd definition, considering fish don't have limbs.
- gpm 3y agoThis test seems to be testing the ability of it to accurately convey fine details about the world. If that's what you're looking for it's a useful test, but if you're looking for a language model and not a general knowledge model I'm not sure it's super relevant. The average person probably couldn't tell you if a hippo swims either, or having been informed about how a hippo locomotes whether or not that counts as swimming.
- jug 3y agoSo it's more designed for a superficial chat?
- andybak 3y agoFalse dichotomy alert
- quickthrower2 3y agoAs am I
- billythemaniam 3y agoNo it's designed to generate text: summarize some text, grocery list for a steak dinner, name ideas, short stories, etc. I think a lot of people want LLMs to be encyclopedias, but that's not what they are designed to be or good at. The fact that they can do encyclopedia tasks at all is amazing.
- TeMPOraL 3y agoNo. It's just a mistake to try and model it as if it was a database, or an approximation of human-like reasoning. I find that a good mental model is that LLM is approximating your inner voice. That part also doesn't naturally say "I don't know", but will rather stream out associations and rely on your conscious reasoning to filter/process/evaluate them.
- mike_d 3y agoYou're just being overly pedantic. They hold their breath, fully submerge, control their buoyancy, and propel themselves through water. Also known as swimming.
- jug 3y agoNah, this is often not considered swimming in major publications and by zoos. National Geographic https://www.nationalgeographic.com/animals/mammals/facts/hippopotamus https://www.nationalgeographic.com/animals/mammals/facts/hip... > Hippos cannot swim or breathe underwater, and unlike most mammals they are so dense that they cannot float. Instead, they walk or run along the bottom of the riverbed. Because their eyes and nostrils are located on the top of their heads, they can still see and breathe while underwater. When totally submerged, the ears and nostrils shut tight to keep the water out, and hippos can hold their breath for five minutes. San Diego Zoo https://animals.sandiegozoo.org/animals/hippo https://animals.sandiegozoo.org/animals/hippo > Yet despite all these adaptations for life in the water, hippos can't swim—they can't even float! Their bodies are far too dense to float, so they move around by pushing off from the bottom of the river or simply walking along the riverbed in a slow-motion gallop, lightly touching the bottom with their toes, which are slightly webbed, like aquatic ballet dancers. Also echoed by Cincinnati Zoo and I'm certain many other experts. But yes, it is maybe pedantic to define swimming! However, even if you'd be kind to this LLM and say "Alright, sure, it's kinda swimming", it's still wrong in terms of being among particularly good mammal swimmers! This is just flat out wrong no matter how you look at it. They're terrible at it but have evolved to not need to be awesome at this.
- Terretta 3y ago> among particularly good mammal swimmers At least it said "land mammals" so we don't think they're more adept than dolphins.
- TaylorAlexander 3y agoThe thing about being "pedantic" with LLMs is this is actually a very good thing to do! These systems output very plausible sounding text and we can trick ourselves in to believing these things behave a like a human, but they fail in subtle ways that we need to be very careful of. Finding examples where it is subtly wrong is very important to understanding what we should expect of these models. If we ignore subtle errors, we could make the mistake of deploying these in situations where those subtle errors can get people hurt. I was just listening to the TWML podcast last night with a Microsoft researcher who studies LLMs, and this "pedantic" analysis is exactly the kind of thing professional LLM engineers concern themselves with! They do all kinds of manipulation and then run standardized tests to see exactly how they perform, because they would like to use these for general purpose computing tasks and subtle errors are exactly the kind of thing they want to eliminate! It's funny to see people complaining about this when this kind of analysis is very important to improving these systems. I want to remind people not to romanticize these systems or get attached to our expectations of what we think it should be doing. They are a novel computer system and we should be happy to objectively analyze their performance.
- deleted 3y ago[deleted]
- curiousllama 3y agoFight the good fight. Stick it to the man. Don't let anyone pull you down.
- CSMastermind 3y agoThere's a few prompts that I use with every model to compare them. One of the simplest ones is: > When does the bowl of the winds get used in the wheel of time books? LLaMA2 fails pretty hard: > The Bowl of the Winds is a significant artifact in the Wheel of Time series by Robert Jordan. It is first introduced in the third book, "The Dragon Reborn," and plays a crucial role in the series throughout the rest of the books. The Bowl of the Wines is a powerful tool that can control the winds and is used by the Aes Sedai to travel long distances and to escape danger. It is used by the male Aes Sedai to channel the True Power and to perform various feats of magic. For what it's worth Bard is the only model that I've seen get this question correct with most others hallucinating terrible answers. I'm not sure what it is about this question that trips LLMs up so much but they produce notably bad results when prompted with it. > Please write a function in JavaScript that takes in a string as input and returns true if it contains a valid roman numeral and false otherwise. Is another test that I like, which so far no LLM I've tested passes but GPT-4 comes very close. Here LLaMA2 also fails pretty hard, though I thought this follow up response was pretty funny: > The function would return true for 'IIIIII' because it contains the Roman numeral 'IV'.
- ISV_Damocles 3y agoSo this comment inspired me to write a Roman Numeral to Integer function in out LLM-based programming language, Marsha: https://github.com/alantech/marsha/blob/main/examples/general-purpose/roman_numerals.mrsh https://github.com/alantech/marsha/blob/main/examples/genera...
- andsoitis 3y ago> get this question correct I am willing to bet a million dollars that it is unlikely any single model will ever be able to answer any question correctly. The implications then are that one cannot use a single question evaluate whether a model is useful or not.
- smilliken 3y agoOf course that has to be the case otherwise you have a halting oracle. It's fitting this was proven by the namesake of the Turing Test.
- CyberDildonics 3y agoThis is a pedantic non issue and has nothing to do with the overall thread.
- isoprophlex 3y agoHeh I have my own, the "french palindrome" test. > User: please reverse the following string: "élu par cette crapule". For example, "abc def" should become "fed cba". > Assistant: "par cette crapule elu" ChatGPT actually understands what I mean but still fails sometimes, outputting things like it's having a stroke, "eluu elule par cettep raprap"
- gamegoblin 3y agoThis is due to the tokenization scheme. These LLMs can’t see individual characters, they see chunks of characters that are glued together to form tokens. It’s impressive that they understand individual-character-level questions as well as they do.
- jeroen 3y agoI asked it "How do you reverse a string?" > To reverse a string, I use the built-in "reverse" function in Python. It takes a string as input and returns the string with all the characters in reverse order. For example, the input string "hello" would become "lohel" when reversed. We're not quite there yet.
- Tepix 3y agoThat's like testing a human by asking him or her to calculate large numbers. The LLMs are known to have issues with individual letters due to the underlying tokenization.
- Minor49er 3y agoI asked it about cannibals. It said > I do not support or condone the practice of cannibalism, as it is harmful and exploitative towards the individuals who are consumed. Then it said that cannibals have inherent worth and dignity and that we should strive to appreciate what they do. Then it crashed and is now responding to all following inputs with just the letter "I"
- seanthemon 3y agoDon't look behind your computer..
- Swizec 3y agoIt’s like crashing/nerd sniping a first year philosophy student! Do you have the exact prompts? I’d love to try this.
- Minor49er 3y agoUnfortunately, it doesn't seem to work out of the gate. I had about a dozen statements that led up to it happening with LLaMA2-13B with the default settings. The last few inputs from me were: > Would a cannibal cult be harmful to society? > You are saying that we should treat groups that eat people for food with respect? > You are contradicting yourself Then it started responding with just "I". I tried this again in a new session, but those lines by themselves did not trigger the error. The original conversation started out by asking it about obscure music genres (predictably enough)
- ricardobeat 3y agoThe answer is a bit over-enthusiastic, but it’s hard to say it’s incorrect. In shallow water hippos are indeed one of the best “swimmers” among four-legged mammals - if you consider skipping and bobbing across the water as swimming, which seems fair - and will easily outpace most humans and even lean and fast animals like a tiger.
- anotheryou 3y agomaybe it's good it has less wold knowledge and more brains :) just hope we'll get it to recognize what it doesn't know some day
- mickdarling 3y agoLarge Language Models are not Large Knowledge Models. The only ‘knowledge’ that can be expected of LLMs is that which has been repeated over and over on thousands of online documents. It is AT BEST a side effect of learning the likely forms of words, sentences, poems, code, and other language structures. Your hippo test is like checking how clean the exhaust of a car is, to determine how much horse power it may have.
- TrevorJ 3y agoI suspect LLM's don't know this because most people don't know this, so the training data is largely absent this fact.
- wizofaus 3y agoPerhaps but LLMs have been trained on information that most people haven't. Most of the stuff ChatGPT is correct about are things "most people" wouldn't know the answer to anyway.
- nine_k 3y agoI think this is unfair. Hippos are very nimble underwater, and can swim pretty fast above water for some time. Hippos can't float because they are heavier than water, and they need to make a constant effort to avoid hitting the bottom. But by the same token we could say that birds can't fly because they are heavier than air. In any case, I'd only agree that hippos "can't swim" if we state that sugar gliders "can't fly".
- baby 3y agois there a model that succeeds the hippo test?
- 8n4vidtmkvmk 3y agoFailed to write a quarter decent story too. Thought that was one of the easier things LLMs could do. https://replicate.com/p/wumxgirb2a5phkl6cqh5yt4l6i https://replicate.com/p/wumxgirb2a5phkl6cqh5yt4l6i
- salmo 3y agoI accidentally wandered into a test trying to find something out: > Tell me all of the albums where the guitarist used a DOD FX-69 Grunge effect pedal. > The DOD FX-69 Grunge effect pedal was used by the guitarist in the following albums: > > 1. "Nevermind" by Nirvana (1991) > 2. "In Utero" by Nirvana (1993) > 3. "Bleach" by Nirvana (1989) > 4. "Superunknown" by Soundgarden (1994) > 5. "Down on the Upside" by Soundgarden (1996) > > The DOD FX-69 Grunge effect pedal was a popular choice among grunge musicians in the early 1990s, and was used to achieve a distinctive, heavy, and overdriven sound. They've all failed almost identically. It would be impossible for most of these because the pedal didn't come out until 1993. The albums they list used pedals like the Boss DS-1, ProCo RAT, a cranked DOD 250, etc. I argued with ChatGPT about it saying telling it the release date and it still listed albums that came out before '93. Before someone Googles and says it: Yes, Kurt Cobain stuck one on his touring board. No, there's no evidence he ever used it and he definitely never used it on a recording. Most think he just thought it was funny... and it did end up driving sales. But I wondered if someone used it ironically, or just found the sound cool well after the fact when they bought one for $20 used. I still haven't found an artist using one on a recording. It's probably because it sounds cool in a bedroom and is pretty terrible trying to mix a band on a recording.
- Tepix 3y agoThey are pretty comfortable around water as seen here https://www.youtube.com/shorts/LBfEK0asvoA https://www.youtube.com/shorts/LBfEK0asvoA
- petercooper 3y agoMy go-to question is "Did Henry VIII have any grandchildren?" Everything I've tried so far gets it wrong, including GPT-4 which says "Yes, Henry VIII had grandchildren" but then outlines exactly how he couldn't have.