5 ms·

> No one here is a scientist and no one treats any of this as science. Where's the criteria for the emprical adequecy of NLP systems as models of language? Spec

by nl 5y ago

> No one here is a scientist and no one treats any of this as science. Where's the criteria for the emprical adequecy of NLP systems as models of language? Specifying any, conducting actual hypothesis tests, and establishing a theory of how NLP systems model language -- this would immediately reveal the smoke-and-mirros. What do you mean? I'm not a scientist but I play one sometimes, and I managed a whole team of them working in this field. The theory of language models is well established. > Where's the criteria for the emprical adequecy of NLP systems as models of language? There are lots(!?) I think the Winograd schema challenge[1] is an easy one to understand, and meets a lot of your objections because it is grounded in physical reality. Statement: The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. Question: Does "they" refer to the councilmen or the demonstrators? The human baseline for this challenge is 92%[1]. PaLM (this Google language model) scored 90% (4% higher than the previous best)[3]. [1] https://en.wikipedia.org/wiki/Winograd_schema_challenge https://en.wikipedia.org/wiki/Winograd_schema_challenge [2] http://ceur-ws.org/Vol-1353/paper_30.pdf http://ceur-ws.org/Vol-1353/paper_30.pdf [3] https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf https://storage.googleapis.com/pathways-language-model/PaLM-... pg 12

mjburgess 5y ago
Indeed, all these test are not of empirical adequacy which really evidences the point. The whole field is in this insular pseudoscientific mould of "its true if it passes an automated test to x%". A theory with empirical adequecy would require you to do some actual research into language use in humans; all of its features; how it works; various theories of its mechanisms etc. And after a comprehensive, experimental and detailed theoretical work -- show that NLP models even *any* of it. Ie., that any NLP model is a model of language. All you do above is design your own win condition, and say you've won. This precludes actually knowing anything about how language works, and is profoundly pseudoscientific. If you set-up tests for toys, and they pass -- good, you've made a nice toy. You may only claim is models some target after actually doing some science.
- nl 5y ago
  A theory with empirical adequecy would require you to do some actual research into language use in humans; all of its features; how it works; various theories of its mechanisms etc. And after a comprehensive, experimental and detailed theoretical work -- show that NLP models even any* of it.* What - specifically - do you mean? There's an entire field adjacent to NLP called Computational Linguistics. Most people in the field work across them both, and there is significant cross pollination. It's unclear if think there is some process in the brain that you think NLP models should be similar to. If this is the case you should look at studies similar to [1] where they do MRI imaging and can see similar responses in semantically similar words. This is very similar to how word vectors put similar concept closely together (and of course how more complex models put concept close together). Or perhaps you think that NLP models do not understand syntactic concepts like nouns, verbs etc. This is incorrect too[2]. [1] https://www.tandfonline.com/doi/full/10.1080/23273798.2017.1323109 https://www.tandfonline.com/doi/full/10.1080/23273798.2017.1... [2] https://explosion.ai/demos/displacy https://explosion.ai/demos/displacy
  - mjburgess 5y ago
    It should do what language does... Language is a phenomenon in, at least, one type of animal. It allows animals to coordinate with each other in a shared environment; it describes their internal and external states; etc. etc. Language is a real phenomenon in the world that, like gravity, can be studied. It isnt abstract. NLP models of language arent models of language. Theyre cheap imitations which succeed only to fool language users in local highly specific situations.
    - nl 5y ago
      > NLP models of language arent models of language. Do you actually know what a NLP Language Model refers to? It literally is a model of the language - it predicts the likelihood of the next word(s) given a set of prior word(s). It seems you think people just throw some data at a neural network and then go wow. It's not like that at all - the field of NLP grew out of linguistics study and has deep roots in that field.

by nl 5y ago

mjburgess 5y ago

Indeed, all these test are not of empirical adequacy which really evidences the point. The whole field is in this insular pseudoscientific mould of "its true if it passes an automated test to x%". A theory with empirical adequecy would require you to do some actual research into language use in humans; all of its features; how it works; various theories of its mechanisms etc. And after a comprehensive, experimental and detailed theoretical work -- show that NLP models even *any* of it. Ie., that any NLP model is a model of language. All you do above is design your own win condition, and say you've won. This precludes actually knowing anything about how language works, and is profoundly pseudoscientific. If you set-up tests for toys, and they pass -- good, you've made a nice toy. You may only claim is models some target after actually doing some science.

nl 5y ago
A theory with empirical adequecy would require you to do some actual research into language use in humans; all of its features; how it works; various theories of its mechanisms etc. And after a comprehensive, experimental and detailed theoretical work -- show that NLP models even any* of it.* What - specifically - do you mean? There's an entire field adjacent to NLP called Computational Linguistics. Most people in the field work across them both, and there is significant cross pollination. It's unclear if think there is some process in the brain that you think NLP models should be similar to. If this is the case you should look at studies similar to [1] where they do MRI imaging and can see similar responses in semantically similar words. This is very similar to how word vectors put similar concept closely together (and of course how more complex models put concept close together). Or perhaps you think that NLP models do not understand syntactic concepts like nouns, verbs etc. This is incorrect too[2]. [1] https://www.tandfonline.com/doi/full/10.1080/23273798.2017.1323109 https://www.tandfonline.com/doi/full/10.1080/23273798.2017.1... [2] https://explosion.ai/demos/displacy https://explosion.ai/demos/displacy
- mjburgess 5y ago
  It should do what language does... Language is a phenomenon in, at least, one type of animal. It allows animals to coordinate with each other in a shared environment; it describes their internal and external states; etc. etc. Language is a real phenomenon in the world that, like gravity, can be studied. It isnt abstract. NLP models of language arent models of language. Theyre cheap imitations which succeed only to fool language users in local highly specific situations.
  - nl 5y ago
    > NLP models of language arent models of language. Do you actually know what a NLP Language Model refers to? It literally is a model of the language - it predicts the likelihood of the next word(s) given a set of prior word(s). It seems you think people just throw some data at a neural network and then go wow. It's not like that at all - the field of NLP grew out of linguistics study and has deep roots in that field.