14 ms·
I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show diff
by hn_throwaway_99 7y ago
I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks.
One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ https://pilehvar.github.io/wic/)
One of their examples, though, didn't make any sense to me:
1. The pilot managed to land the airplane safely
2. The enemy landed several of our aircrafts
It says that the word "land" does NOT mean the same thing in those sentences. I am a native English speaker, and I honestly don't understand what they are thinking the second sentence means. Shot them down? If so, I have never heard "landed" used in that context, and it appears neither has Merriam-Webster. Also, the plural of aircraft is just "aircraft", without the s.
- mooman219 7y agoThis is definitely where the 10.2% of human failures are.
- matthewowen 7y agoI think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. For me, my first read of the sentence would definitely be that it means shot down.
- otterpop 7y agoso this is why the human score is 89.8 :)
- hn_throwaway_99 7y agoAhh, just found an example where that's taken from https://glosbe.com/en/en/land https://glosbe.com/en/en/land. If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". I have still never heard landed used in that way, and again in other dictionaries I searched I couldn't find that definition either. Thus, this is a case where the "AI" may get it "right", and me, the human would get it "wrong", but that still feels like it's missing a huge point. It feels you could get a number of errors by the human which the AI gets "right", but in fact the human is better able to detect what is rare, uncommon or at least ambiguous.
- rhizome 7y ago>If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". They're not shy about illustrating a military application up front!
- mike00632 7y agoI've worked in aviation for 8 years and also didn't understand this use of "landed". I've heard "grounded" used like this: "The maintenance issues gounded the jet," but not "landed".
- longblack 7y agoI think the sentence is referring to aircraft that have been forced to land by the enemy, in contrast to "grounded" aircraft that had not taken flight. I haven't worked in aviation so my understanding of terminology could be wrong, but either way it is definitely an unusual example.
- mike00632 7y agoRight, I think you understand the word as I do: 'verb' + ed. "The enemy landed the jet" as in they forced the jet to land either directly or indirectly. This would mean that the two sentences use "landed" the same way. But my understanding is SuperGLUE's offical answer is that these use "landed" differently with the rational that "landed" is idiomatic and just means to procure or bring about (e.g. "I landed the job") and it happens to be used with planes.
- theaeolist 7y ago> I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. My buddy is a pilot and they always say "I landed the takeoff pretty good. PRETTY GOOD!"
- nicoburns 7y agoI think they might going for 'landed' as in 'landed a deal'. Maybe?
- somebodythere 7y agoIn option 1, the aircraft met the ground gently and safely. In option 2, the aircraft met the ground violently and lethally.
- p1esk 7y agoNot necessarily. Maybe they captured it.
- im3w1l 7y agoThe only possible explanation I can think of is this. 3. a : to catch and bring in // land a fish b : gain, secure // land a job landed the leading role imagine enemy soldiers capturing a base or hangar ship including the aircraft. It's kind of a stretch though.
- rahimnathwani 7y agoFor #2, my immediate read was that the planes had been shot down. If the context were to suggest that the enemy had somehow hijacked the planes, then of course the word land would mean the same in both sentences. I have never used or heard 'land a plane' in this context, but the sentence didn't immediately strike me as unnatural, incorrect or unclear.
- taneq 7y ago> I have never used or heard 'land a plane' in this context, but the sentence didn't immediately strike me as unnatural, incorrect or unclear. It struck me as pretty awkward and very ambiguous. It probably means 'obtained' but 'captured' would be a far better word in that case. The suggestions that it means 'hit/shot' don't work because in that case it's not the aircraft that is landed but the shot, which is landed on the aircraft. Also the use of the incorrect plural "aircrafts" when 'aircraft' is both singular and plural makes me think it's just a poor question. The very fact that there's so much discussion about it is evidence that it's not straightforward even among native English speaking humans.
- goldenkey 7y agoTime matters too. Current tech would hint to us that the planes had been shot down. But in the future that sentence might mean hacking and theft of the actual planes, an actual landing.
- bobbyi_settv 7y agoI've never seen "landed" used as in the second sentence, but I was definitely able to understand from context that it was not being used to mean the same thing as in the first sentence.
- archontes 7y agoYou've never landed a fish?
- hossbeast 7y agoHave caught a fish though
- cbetti 7y agoIs land an acceptable habitat for a fish?
- tremon 7y agoIs "landing a fish" the same thing as "watering a plant"?
- Frondo 7y agoI haven't, though I'm familiar with that use of "landed" for fish. As a lifelong native speaker (PNW English), I've also never heard "landed" used to refer to shooting down or capturing enemy airplanes. I could understand it from context, which is what I suppose the software is also going for, but I'd mark it with a red pen if someone showed me that sentence, just for clarity's sake (i.e. understandable from context but should be replaced).
- randcraw 7y ago'Landing' an aircraft does not imply shooting it down. 'Downing' an aircraft does imply that. These uses of 'land' and 'down' are military euphemisms for the use of force to compel a reluctant pilot to land. The difference is the degree of violence used. Involuntary 'landing' implies the aircraft is forced to land by a party other than the pilot because if the pilot did not comply the plane would be shot down or collide or crash. It usually implies survival of the pilot. 'Downing' also means involuntary removal of the aircraft from the sky, but does not denote that a violent landing did occur, only that the likelihood of violence is much greater because a (more abrupt) landing was forced upon the pilot. From what I've read, 'downing' usually implies the plane crashed.
- mc32 7y agoYeah I don’t get the s at the end of aircraft. Landed, it would seem, would be land as in acquire, although that’s a bit odd of a construction. It seems rather forced. It may possibly mean that the aircraft were forced to land by the enemy. So it’s a tortured construction.
- eindiran 7y agoThe second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).
- LaMarseillaise 7y agoIf taking the "captured" interpretation, I think it could be reasonably inferred that they successfully landed the aircraft at an airfield afterwards (same meaning). This was my initial read of it and it does not seem strange to me on reflection. I would like also to point out that even if we do interpret the second as meaning "destroyed", the first could then be interpreted as a combat aviator shooting down an opposing aircraft, bringing us back to the same meaning. Or perhaps both of my interpretations are correct and the meanings are different... What this tells me is that the benchmark is not very useful.
- TylerE 7y agoLanded in the sense of a fisherman landing a marlin.
- nmeofthestate 7y agoSo at the end of the process they were in possession of the enemy aircraft. Maybe they jumped across in mid-air and wrestled it off the other pilot.
- parksy 7y agoThe benchmark is useful primarily because it puts humans and computers on a level playing field. Human readers will misinterpret written language, and human writers will poorly represent concepts. The propensity to make mistakes in comprehension is unavoidable, humans only approach 90% accuracy, and computers are getting close to the same level of accuracy on the same base materials as humans. The other way of testing would be to devise a test where there is only a single interpretation, where the context is clear, and there is no ambiguity in meaning. In that case a competent human and computer algorithm could be expected to answer all questions perfectly. The purpose of this benchmark on the other hand is to test comprehension when meaning is not explicit and context clues are implied, something humans have had the advantage at over computers until quite recently. The computer won't be 100% accurate, but that's not the purpose of this test.
- mike00632 7y ago2. Reminds me a of a theory that Iran landed an American stealth drone by sending spoofed signals.
- jobigoud 7y agoStill ambiguous. Landed as in make it contact the ground or landed as in obtain, like in landing a job? For me taking an airborne object and making it touch the ground is pretty much the same meaning whether it's from the inside or remotely or shooting it down.
- mike00632 7y agoYes. I think "ambiguous" is the best word to describe all of this.
- infradig 7y agoSame as "to land a punch". To successfully hit a target.
- seanwilson 7y ago> One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ https://pilehvar.github.io/wic/) Can anyone explain what makes this difficult for a machine? What existing knowledge does the machine start with? At a glance, it doesn't feel like it should be difficult if the machine had a large corpus to train on that showed many examples of each words in different contexts.
- billforsternz 7y agoGood point and most of the replies ignore the key point to me which is; You are right about the plural of aircraft and the benchmark is horribly wrong, so why should we take any notice of this benchmark?
- tanilama 7y agoIt means 'succeeded in shooting down' right? Seems pretty contextual, but understandable.
- vonseel 7y agoSeems like a really odd way of saying it but that’s what I’d think too, as in “landed their shots”. This is either a poor question, or a really great question, if the goal of the test is to confuse computers where a human would normally say “huh, weird way of saying that but I guess they mean...”.
- didibus 7y agoIt's weird, because I understood the second one as meaning shoot down, yet to me that's the same definition of landed. You just assume the enemy didn't land them gracefully without a scratch, because they are well, enemies. So I would have answered that the word meant the same thing.
- anonytrary 7y agoThe second implies that the aircrafts were shot down; the first states that the aircraft landed safely. It looks like this reduces to the machine being able to figure out whether or not something is good or bad for the speaker.
- mannykannot 7y agoFrom the abstract of the associated paper: "performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research." It occured to me that hn_throwaway_99's question, and the responses to it, is the sort of dialog in which one could find additional headroom for further research into natural language understanding. We can understand, for example, that while the two uses of 'landed' are different, they are not completely unrelated, and we can explain how they are related, for example by introducing a third construct, 'landed a fish', as a couple of replies have done.
- briga 7y agoLimited headroom? Seems like they're assuming greater-than-human language ability is just impossible and will never be surpassed.
- mannykannot 7y agoI do not think this result is that close to a greater-than-human language ability in general, and I do not think they are claiming it. I think the point is that, with scores on this test closely approaching average human scores, there is not much headroom for this particular test to drive, or measure, further progress.
- ars 7y agoIt's a reasonable assumption if only for the simple reason that humans said the sentences being tested, so how would you surpass that?
- jobigoud 7y agoYou create a new test designed by your newly better-than-human language experts.
- wyattpeak 7y agoI'd argue that greater-tham-human language ability is by definition useless. Language is specifically a human communication tool, there's no value in surpassing the language skill that humans have, if indeed such a thing is even meaningful (what does it mean to be better than the best* French person at French?) * By whatever language-related metric
- _II__II_ 7y agoThe example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.
- andrewstuart2 7y agoOne of my favorite examples that I heard in a David Rock talk which I can no longer find on youtube: "Time flies like an arrow": Time moves swiftly and in one direction. Record the speed of flies in the same way you would an arrow. Time flies, which are a kind of fly, are fond of an arrow. (e.g. Time flies like an arrow, fruit flies like a banana).
- changoplatanero 7y ago"I eat my rice with butter." could mean that you use butter as a utensil to eat your rice with. There is often an unlikely way of parsing the sentence that gives an alternate meaning. The point is to test the computer to see if it can distinguish the likely parse from an unlikely one.
- nl 7y agoThese aren't really alternate _parses_ though (in the sense that they don't give different parse trees). They do highlight the different possible meanings of "with" though. I think "I eat my rice with chicken" vs "I eat my rice with children" vs "I eat my rice with chopsticks" is the canonical example here. There's a whole field in NLP involved in showing what changes happen to entities mentioned in a sentence as a a side effect of the sentence, and this example shows it pretty well.
- changoplatanero 7y agoWouldnt those be different parse trees? Like the "with X" could either be attached either to the verb or the noun
- natch 7y agoSo many options for sentence number two. - The enemy stole the aircrafts, and after some drama in flight managed to land several of them. - The enemy used remote control to force them to land. - The enemy used coercive force to force our pilots to land them. - The enemy captured them. - The enemy shot them down. - During a friendly event while we set our differences with our enemy aside and agreed to fly each other's aircraft at an airshow for some reason, we landed several of theirs, and they landed several of ours. - There was a hearing mistake and "energy" (as in energy beam beamed by a UFO) was accidentally transcribed as "enemy." - The writer is just screwing with us. - The writer is not a native speaker of English, and they made a mistake and actually meant that the enemy boarded several of our (parked) aircrafts. - The writer is creative with language and believes that it would be cute to say that when an enemy projectile struck one of our aircrafts, then the enemy has "landed" that aircraft as one would land men on the moon or land rovers (no pun intended) on Mars.
- mcphage 7y ago> then the enemy has "landed" that aircraft as one would land men on the moon or land rovers (no pun intended) on Mars Or perhaps as one would land a punch.
- ethbro 7y ago- An ML algorithm from the future traveled back in time, writing specific SuperGLUE examples to poison AI research, thereby preventing the emergence of a competitive AI which would also master the secrets of closed timelike curves
- solotronics 7y agoActually the algo was able to determine we exist in a simulation and perform meta programming by hacking the sim infrastructure (higher order dimensions of spacetime) and rewriting the future which to us appears that it traveled to the past.
- jcims 7y agoIn looking through many of the replies to this downstream, it appears that the system is actually correct in that there's an obscure use of 'land' at play in the second sentence. It makes me think that there's going to be many adversarial examples of text that humans parse one way because of common usage while machines parse another way because of details like this.
- devin 7y agoColorless green ideas sleep furiously! Search for it if you’re interested in its origin.
- brainless 7y agoI think the difference in these sentences is about the way to land. In sentence 1, the pilot of the aircraft is in control. In sentence 2, the pilots are not in control, the enemy forced them to land (whatever the means). If I read these two sentences in context of some news, they would evoke very different "landing" scenes in my head.
- mohaine 7y ago#2 is the same as landing a fish. i.e. to place on land what doesn't belong on land.
- gvhst 7y agoIt sounds like the “landed” in 2. is similar in usage as “landed” used in the turn of phrase “he landed the deal.”
- ValleZ 7y agoI'm not a native English speaker and it is pretty obvious to me what both sentences mean.
- Aperocky 7y agoThe example looks like they're not written by native English speaker. It's funny reading English tests from other countries that are not English speaking because a lot of it focus on pedantics that are long lost while following a convention that would to us just feel _different_.
- scarletmantis 7y agoI understood 'landed' as an euphemism for 'shot down'.
- alex_young 7y agoThe plural of aircraft is aircraft. Not aircrafts. https://www.grammar-monster.com/plurals/plural_of_aircraft.htm https://www.grammar-monster.com/plurals/plural_of_aircraft.h... Maybe the _examples_ for a language test should be grammatically correct?
- BiasRegularizer 7y agoTrue, but having some noise in the label is actually good for generalization. If it's only learned on perfectly correct sentences then its tolerance for mistakes will be very low.
- Al-Khwarizmi 7y agoIt depends on what your goal is. But in most cases, I'd say no. If the goal has anything to do with understanding real language written by real humans, it's better for the system to be able to handle texts with errors.
- jasonlotito 7y agoMaybe the examples for a language test should use language that people actually use every day.
- rladd 7y agoMy mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age). She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct. She had to outguess the test creators and answer the questions wrong -- in the "right" way. This seems like a similar situation.
- conjectures 7y agoThis does seem like the meta solution to most tests, particularly standardised tests :)
- qubex 7y agoParaphrasing Simonyi: “Any test you can pass, I can pass meta”.
- Huppie 7y agoI've had the 'pleasure' of taking some 'Microsoft certifications' at various companies I worked at in the past and this sounds extremely familiar. "I probably won't ever do it like that and/or there's a syntax error in all four of the answers... but this is the answer you want to hear. It's wrong, mind you, but it's what you want to hear."
- zerkla 7y agoYep! You have to do away with conventional logic and ask yourself "What insanity would Microsoft recommend I do?"
- Huppie 7y agoIt's not always insanity, sometimes just sub-optimal / way over-engineered in my opinion. They're getting better at it though. More recently I've done their devops certification and it looks like they're recommending somewhat more sane practices now... There were still questions where even after three or four tries at certification / reading up on whatever Microsoft thinks is 'good' we didn't find 'the correct answer' according to Microsoft though... ¯\_(ツ)_/¯
- qroshan 7y agoI landed this job
- throwaway1777 7y agoThe fact this comment sparked so much discussion with some agreeing and some disagreeing says to me that Google did about as well as a human.
- option_greek 7y agoProbably: The enemy grounded several of our aircrafts
- deleted 7y ago[deleted]
- NicoJuicy 7y agoThat's probably why humans have 89,9% and not 90,1% :p
- mrmonkeyman 7y agoShe landed a job. Got access to, acquired, etc.
- gradstudent 7y agoPerhaps the enemy obtained several of our aircraft. In the same sense as one might land a new car in a contest.
- nothrabannosir 7y agoPossible, but also the worst context to use land in. You land a car, but if the game host would say you landed a small airplane, there’d be a laugh from the crowd.
- throwaway936482 7y agoI think this is really interesting, because "the enemy landed several of our aircraft(s)" is the sort of sentence I'd have hauled a student up for using as a teacher, because 1) it's a none standard, arguably incorrect usage they've used either because they're a none native speaker or because they're trying to be clever and failing, and 2) because the plural of aircraft is aircraft. Nevertheless the author of this sentence almost certainly meant land to mean something different (shot down) than the author of the first, and we can infer the author's intended meaning despite the none standard usage. This poorly written sentence is the sort of thing you see all the time in the real world, especially from none native speakers, children, and people writing about a topic outside their expertise. If a program can spot the difference in the usage of the word land between these two sentences and infer what the intended meaning in the second sentence is, then it's doing pretty well. Just inferring that land is used to mean something different in the two sentences is less impressive but still pretty cool and I'm not sure which claim is being made.
- HereBeBeasties 7y agoIf you teach others English, please learn the difference between "none" and "non". You mean "non-standard" in all your examples here (if British) or perhaps "nonstandard" (if American).
- scandox 7y agoWell "he landed the deal" implies a score or a hit. So to say they "landed" the planes could vaguely make sense but it is hardly good English. They might have been thinking of "grounded"?
- randcraw 7y ago'Grounded' means the plane could not take off. It was on the ground and must remain there. Landing a deal (or a fish) is like landing a plane. A human acts to cause a desired outcome. Unlike forcing a pilot to involuntarily land a plane, the perspective of the fish as involuntarily being forced to land is not a necessary inference for this use of 'land'. Geez, language can be subtle.
- mrosett 7y agoI believe it’s “land” in the sense of “land a fish” (or a prize in general) which is a less common but legitimate usage.
- SiVal 7y agoYes, the second should be, "The enemy downed several of our aircraft." Landed can be used to mean "bagged," as in, "We finally landed the Smith account," (it's a fishing term), but it should not be used in this figurative sense when referring to aircraft, because of the obvious confusion with the common, concrete sense of the word. And, yes, it should be aircraft.
- m3at 7y agoI think people are digging too deep for an answer here... it seem to me to be a simple mistake, which on the scale at which they're evaluating those models is not statistically significant.
- dmurray 7y agoIt's being used by analogy with "landing a fish". I've never heard it either, but I could believe it's in the argot of military airmen in some English-speaking country.
- main5tream 7y agoIt's conceptually the same - having an entity go from water or air to the the ground. The hard part would be to associate the fact that there's no way for an 'enemy' to land the aircraft other than to do so forcibly which implies shooting it down.
- jasonlotito 7y agoSo, here is the thing. ML shouldn't just be about learning rules. It should be about actually learning, and understanding. Just because you've never heard the word used that way, you were able to infer it meant something different. Even with the use of aircrafts. We all make mistakes when writing or speaking. We don't let that get in the way of interpreting the information being passed. Even if we post comments that contain errors.
- ebg13 7y ago> I am a native English speaker, and I honestly don't understand what they are thinking the second sentence means Clearly the enemy conferred lesser nobility and commensurate landownership unto said aircrafts. https://en.wikipedia.org/wiki/Landed_gentry https://en.wikipedia.org/wiki/Landed_gentry
- randcraw 7y ago1) The pilot [voluntarily] brought down his aircraft. 2) The pilots [involuntarily] brought down their aircraft [because some authority figure(s) forced them down.] The active verb 'land' can be performed by different actors: pilot vs a more powerful agent (usually who flies an armed aircraft). The voluntary/involuntary agency is a subtle difference that only those familiar with this military practice are likely to grok.