7 ms·
Do Language Models Know How Heavy an Elephant Is?
- unityByFreedom 6y ago> We also come up with a new version of BERT, called NumBERT, with improved numerical reasoning by replacing numbers in the pretraining text corpus with their scientific notation, which more readily exposes the magnitude to the model, and demonstrate that NumBERT representations capture scale significantly better than all those previous text representations. That's really cool. I like the writing in this post. Concise and dense with info.
- BlahGod420 6y agoIt's a data problem. Not a language model one. Wolfram alpha will literally answer.
- notum 6y agoI tried "How heavy is half an elephant on Moon?" and all I got is a wiki-like page :/
- wyattpeak 6y agoThat's a heck of a question to expect an AI to answer. It's weird (we almost never discuss half animals), oddly phrased ("on the moon" is much more standard), and runs afoul of an imprecision in English (we normally given masses when asked about weight, this requires an answer in force, or else by analogy a mass with equivalent Earth-weight).
- notum 6y agoIt's tailored to be difficult, I took this course right after the question above was autosuggested to me. Also while "the moon" could mean anything, there's only one Moon. I gave it a fighting chance.
- IshKebab 6y agoDunno if you're native but ".. on Moon" is not right. It should be "..on the Moon". Don't ask me why it is different for Earth and the Moon but it is! I would expect any language model to be robust to minor errors like that though so I doubt it makes any difference.
- nandhinianand 6y agoBleh.. .English and it's weird grammar .. Sometimes I find it such a messed up language compared to my native tongue..... I guess it's the separation of modifiers and nouns that's part of the problem.
- nandhinianand 6y agoSeriously a downvote ?? for what?? ughh.... "bavula kappa" ==> frog in the well.. a metaphor for biased people(or people who live in a bubble).
- notum 6y agoThanks! I'm indeed not native :) In my own defense it "felt" like it was missing something. However I am confident there are many moons and just one Moon, our own. Also, the answer is 1.62kN to 4.8kN, I know Wolfram doesn't employ a language model or anything of a sorts, but with all the other NLP magic I've seen I sort of expected a valid answer.
- astrange 6y agoIt's because the name of the moon is "The Moon", not "Moon". I think SF writers sometimes pretend it's called "Luna" so it'll have a more interesting name.
- nicoburns 6y agoIt is "la luna" in spanish. So I don't think SF writers are pretending.
- ForHackernews 6y agoIt is a tough question, but we keep hearing from AI-boosters about how the Singularity is right around the corner and have you seen this AI that can make plausible-looking text?
- TheCoelacanth 6y agoIt's a question that a B+ student in high-school physics could answer with a few minutes of googling. If that's beyond the capabilities of AI, then it obviously isn't living up to the hype.
- wyattpeak 6y agoWell, yes. A B+ high-school student can also reliably recognise the number 710. If you think that AI should be able to do something just because a high-school student can, I think it's your expectations which are out of touch.
- TheCoelacanth 6y agoMy expectations are fine. The AI hype is out of touch.
- perl4ever 6y agoA question that's not "weird" has been asked and answered before, so you can answer it with a database and zero "intelligence". I think you're lowering expectations for an AI to zero.
- 6pac3rings 6y agoNeeds a supervisor for cheeky monkey curve ball playful question to interpret answer on gradient of seriousness from thousands of mind model with access to knowledge library and language as commonly used. The part before "on" doesn't sound out of place from a template for language exchange in a butcher's shop and "on Moon" is a playful reframing adjusting the physics model. A child lucky to have a parent with training in physics would get an answer easy to this question.
- notum 6y agoOr it simply needs to be augmented with something other than GPT. The form of my question is very easily solvable in a programmatic way, just not by a GPT based neural net. Someone mentioned multi model training, that sounds awesome!
- hypertele-Xii 6y ago> (elephant weight) / 2 * (moon gravity / earth gravity) > 160 to 520 kg There you go. Wolfram Alpha literally answered, just needed to translate your question to math ^_^
- notum 6y agoGood effort! But gram is not a unit of weight.
- Cybiote 6y agoA quibble is that when I see kg, mass, I expect that it should be measured the same whether on Earth or on the moon. The answer should be in Newtons. Our query to Wolfram Alpha should thus be: (mass of elephant)/2 * moon gravity
- mannykannot 6y agoNice answer, but it actually supports the point being made here. NLP is an attempt to see how far we can go in processing natural language, without doing this sort of translation, and without structured knowledge bases. As (I'm pretty sure) Wolfram Alpha contains a structured knowledge base and draws on it for this sort of question (though not without help in this case), its abilities are orthogonal to the issues addressed in the article.
- charcircuit 6y agoHalf an elephant is undefined. Which half are you counting? Each half doesn't necessarily have the same weight.
- notum 6y agoActually "half" is pretty dang well defined :)
- rsfern 6y agoWell even if you take the most symmetric halves by volume, the density of an ideal elephant is not uniform. Some organs don’t come in pairs and all that I guess it depends on how many hairs you want to split and how precise you want the answer
- notum 6y agoThe answer can never be precise for the very reason that not all elephants weight the same (breed, age etc.). I was looking for an average from the get-go, but a half is a half!
- rsfern 6y agoI agree with you that a reasonable answer is an average, but a half on a volume basis is not necessarily equivalent to a half on a mass basis
- Engineering-MD 6y agoHowever most unpaired organs don’t contribute to a mass asymmetry. It’s thought this has evolved to aid with locomotion as any asymmetry will negative impact on efficiency
- charcircuit 6y agoI disagree. You can have a left half and right half, but also a front half and rear half. Half of the weight of an element would be well defined since it is just a number.
- gopkarthik 6y agoIt was interesting to note that on typing "how heavy", the only search suggestion I got was "how heavy is an elephant".
- unityByFreedom 6y agoEvaluating the quality of language models is a challenge in itself. This post just presents another way to see how much your model can understand alongside a new model. This is typical when presenting new tech for which old evaluation methods might not tell you anything. It's not really about getting the answer to that question as it is about figuring out how much information your model can glean from text.
- ilaksh 6y agoI feel like it's fairly well-established that these text-only models cannot not have the capability of common sense reasoning because they don't incorporate sensorimotor/spatial/temporal/etc. knowledge. They can't because they are not trained with any of that, just text.
- madsbuch 6y agoMulti modal learning is the new kid on the block.
- unityByFreedom 6y agoRead the post. It's an incremental improvement on existing language models to incorporate numerical understanding. It's more than just the title.
- ilaksh 6y agoI did read the post. I guess I should have been more explicit. These types of incremental improvements seem to have limited utility and researchers who want to improve the models substantially may find more payoff in pursuing the integration of other types of data.
- unityByFreedom 6y agoIt's a clever technique, not data, so anyone can incorporate it. As for improvement the post shares its metrics and anyone could experiment on their own tasks to see if it brings improvement or not.
- nothis 6y agoI find it interesting how hard it is to think of a question that truly cannot be answered by just reading lots of text. Stuff like "how can you carry water without a vessel?" (you freeze it!), which might stump a lot of AI models. But then you think... there should be texts that explain that ice is water and that ice can be carried. Similarly, the problem with numbers (lenghth, height, etc) isn't a lack of data, it's messy notation which is still easily parsed by human readers and probably could be solved. If anything, I've become more convinced that text alone has enough information to solve some eerily specific problems. It's quite a powerful tool.
- jeffreygoesto 6y agoTL;DR: We did a lot, it improved a bit, but no.