7 ms·

OK so most of the comments seem dismissive, picking on unfortunate statements (between Newton & Zuck) or an ungenerous reading of the claim. I'll take the othe

by netcan 5y ago

OK so most of the comments seem dismissive, picking on unfortunate statements (between Newton & Zuck) or an ungenerous reading of the claim. I'll take the other side. Consider psychology, economics, ecology, climate, many health & biology related fields, etc. These fields have struggled to succeed within the Popperian/Newtonian framework. We haven't found an F=ma or E=mc^2 that is fundamental, predictive and generally useful. This has resolved to a lot of frustrating modeling and unfalsifiable theories. P-hacking, replication crisis & such. Marginalist economics is a great example. If you squint, it kinda conforms to Popperian models. In practice though, it's not workable. In practice, it's riddled with post fact storytelling, subjectivity and opinion. The "high science" mode of enquiry is amazing. It has arguably advanced us more as a species than any other idea. It is, however, not the only mode of enquiry. We walk around the world forming tentative theories for how the world works. They're generally not simple, expressible or general. They are, however, predictive. Jane knows how Joe will react to X. She doesn't know it with certainty or accuracy, but she's got pretty high chances of being right. Call that a theory, a model... whatever. It's a real way of having knowledge. Farmers' understanding of the farm. There's no guarantee that the phenomenon we're interested in will be expressible in an F=ma manner. Complexity exists.

yldedly 5y ago
>There's no guarantee that the phenomenon we're interested in will be expressible in an F=ma manner. Complexity exists. That's a fair point. But having a good theory doesn't necessarily mean that it's expressible as a succinct equation. The theory of how metabolism works is insanely complicated, but it's a good theory nonetheless, because every part of it plays a functional role, it explains a wide range of phenomena and it's logically compatible with other successful theories. None of that holds for deep learning models.
- netcan 5y ago
  Good points. Besides being an overly provocative term, I don't think "post-theory science" is a belligerent to "theory-based" science. It's not logically compatible or incompatible with other theories, because it's not logically based. It may be contradict theoretical predictions, and that's where empiricism comes in. Long term, if successful, deep learning models of metabolism may be useful or irrelevant to the development of a theoretical understanding.
naasking 5y ago
> This has resolved to a lot of frustrating modeling and unfalsifiable theories. P-hacking, replication crisis & such. It's entirely possible we haven't found such models because of the p-hacking, poor replication and publications with low signal to noise ratios, and not vice versa. Psychology's replication rate was in the mid 30% range. It would be hard to see any general trends with such unreliable data.
smoyer 5y ago
This is a great comment and what holds a lot of today's AI back is indeed complexity. The only question I would add is whether it's still science (hypothesize, predict and test as stated) if it's not a human performing the expected steps - e.g. is training a neutral network through feedback iterating through these steps (with short cycle times?)
- musingsole 5y ago
  Probably depends on how you define science. If you put science on a spectrum with it on one end and magic on the other (so the spectrum is more of explainable knowledge versus useful but inexplicable rituals), AI definitely moves towards the magic end of the spectrum and maybe into a field commonly called "engineering" for the fact that it bastardizes knowledge with utility.
  - smoyer 5y ago
    My question above would definitely not extend to the word engineering (and by association I probably have to withdraw the idea that AIs could do science.)
- netcan 5y ago
  >> The only question I would add is whether it's still science (hypothesize, predict and test as stated) if it's not a human performing the expected steps This is possibly just a semantic question. If so, I have no problem calling it something else.
- PaulDavisThe1st 5y ago
  > If it's not a human performing the expected steps Nothing is performing those steps in a NN.
- marcus_holmes 5y ago
  this is my question with a lot of this "science" - how are we ensuring that we're not incorporating researcher biases when setting up and training a new model? To take a reasonably provocative example: if a new climate model predicted extremely low rates of change, would it be tuned so it showed change in the accepted range? If so, then how useful is this model to science?
RandomLensman 5y ago
A lot things in practice are using empirical methods and are not looking at F=ma like representations. The basic issue with going just empirical is that there is no way of knowing if the relationship actually exists or if what is observed is a symptom of something else. (Or, in the extreme case: random. Some bits in finance have that problem, for example, finding an edge in markets). Most successful application of ML etc. are not relying in areas where there are no ideas about any relationships, but rather where there are a lot of confounders and the relationships are complex/difficult to formulate. Sticking with on of the examples, we know that proteins fold in somewhat predictable ways, because we do have theories around the "mechanics" there. So good applications are not strictly in the realm of scientific theory but rather in being able to apply and use such theories in previously inaccessible ways.
- PaulDavisThe1st 5y ago
  > Sticking with on of the examples, we know that proteins fold in somewhat predictable ways, because we do have theories around the "mechanics" there But we were broadly unable to leverage those theories and knowledge into a predictive system. AlphaFold largely dropped all that (IIUC), and works much better without it. (I was once a PhD candidate in protein folding, and left in part because I felt the field was unlikely to make any substantive progress. It didn't, for at least 30 years, until NN-based "modelling" started to replace all approaches up till then)
  - dataangel 5y ago
    Well, but wasn't AlphaFold at least in part trained based on the existing mechanical models? Or was AlphaFold only trained on ground truth imaging of the folding process?
    - PaulDavisThe1st 5y ago
      It was trained on experimentally-determined physical structures. Not predicted models based on various folding engines.
  - 5y ago
eru 5y ago
Not sure why you attack economics? It works really well to eg predict and understand the impact of tariffs or taxes or deregulation. Yes, economists can't predict recessions in advance. And economics also tells you exactly why: the efficient market hypothesis tells you that if people knew that stock prices were going down next year, prices would already react to that knowledge today.
- echopurity 5y ago
- netcan 5y ago
  It's not an attack.
  - eru 5y ago
    OK! I might have misread your comment.
- rkk3 5y ago
  > Yes, economists can't predict recessions in advance Aren't they actually pretty good at predicting recessions? Thought the joke is that they predicted 12 of the last 7.
- convolvatron 5y ago
  pretty sure the parent is doing the exact opposite. other forms of inquiry aren't valueless because they aren't falsifiable.
  - netcan 5y ago
    cheers. Dead on.
snowwrestler 5y ago
You can create a complex system to make predictions about complex systems. But without underlying theory, you can’t put bounds on its accuracy or utility. Such a system will work right up until it doesn’t, with no way to predict in advance where that point is. We see this with people all the time. Jane thinks she knows how Joe will react to X but Joe does something unexpected. A system that makes accurate predictions an unpredictable percentage of the time is not that useful. But it might be great for fooling ourselves or others.
- kybernetikos 5y ago
  Jane being able to predict how Joe will react most of the time turns out to be super useful in real life. Sure it fails sometimes, but nobody would voluntarily give up that ability.
deleted 5y ago
[deleted]

by netcan 5y ago

yldedly 5y ago

>There's no guarantee that the phenomenon we're interested in will be expressible in an F=ma manner. Complexity exists. That's a fair point. But having a good theory doesn't necessarily mean that it's expressible as a succinct equation. The theory of how metabolism works is insanely complicated, but it's a good theory nonetheless, because every part of it plays a functional role, it explains a wide range of phenomena and it's logically compatible with other successful theories. None of that holds for deep learning models.

netcan 5y ago
Good points. Besides being an overly provocative term, I don't think "post-theory science" is a belligerent to "theory-based" science. It's not logically compatible or incompatible with other theories, because it's not logically based. It may be contradict theoretical predictions, and that's where empiricism comes in. Long term, if successful, deep learning models of metabolism may be useful or irrelevant to the development of a theoretical understanding.

naasking 5y ago

> This has resolved to a lot of frustrating modeling and unfalsifiable theories. P-hacking, replication crisis & such. It's entirely possible we haven't found such models because of the p-hacking, poor replication and publications with low signal to noise ratios, and not vice versa. Psychology's replication rate was in the mid 30% range. It would be hard to see any general trends with such unreliable data.

smoyer 5y ago

This is a great comment and what holds a lot of today's AI back is indeed complexity. The only question I would add is whether it's still science (hypothesize, predict and test as stated) if it's not a human performing the expected steps - e.g. is training a neutral network through feedback iterating through these steps (with short cycle times?)

musingsole 5y ago
Probably depends on how you define science. If you put science on a spectrum with it on one end and magic on the other (so the spectrum is more of explainable knowledge versus useful but inexplicable rituals), AI definitely moves towards the magic end of the spectrum and maybe into a field commonly called "engineering" for the fact that it bastardizes knowledge with utility.
- smoyer 5y ago
  My question above would definitely not extend to the word engineering (and by association I probably have to withdraw the idea that AIs could do science.)
netcan 5y ago
>> The only question I would add is whether it's still science (hypothesize, predict and test as stated) if it's not a human performing the expected steps This is possibly just a semantic question. If so, I have no problem calling it something else.
PaulDavisThe1st 5y ago
> If it's not a human performing the expected steps Nothing is performing those steps in a NN.
marcus_holmes 5y ago
this is my question with a lot of this "science" - how are we ensuring that we're not incorporating researcher biases when setting up and training a new model? To take a reasonably provocative example: if a new climate model predicted extremely low rates of change, would it be tuned so it showed change in the accepted range? If so, then how useful is this model to science?

RandomLensman 5y ago

A lot things in practice are using empirical methods and are not looking at F=ma like representations. The basic issue with going just empirical is that there is no way of knowing if the relationship actually exists or if what is observed is a symptom of something else. (Or, in the extreme case: random. Some bits in finance have that problem, for example, finding an edge in markets). Most successful application of ML etc. are not relying in areas where there are no ideas about any relationships, but rather where there are a lot of confounders and the relationships are complex/difficult to formulate. Sticking with on of the examples, we know that proteins fold in somewhat predictable ways, because we do have theories around the "mechanics" there. So good applications are not strictly in the realm of scientific theory but rather in being able to apply and use such theories in previously inaccessible ways.

PaulDavisThe1st 5y ago
> Sticking with on of the examples, we know that proteins fold in somewhat predictable ways, because we do have theories around the "mechanics" there But we were broadly unable to leverage those theories and knowledge into a predictive system. AlphaFold largely dropped all that (IIUC), and works much better without it. (I was once a PhD candidate in protein folding, and left in part because I felt the field was unlikely to make any substantive progress. It didn't, for at least 30 years, until NN-based "modelling" started to replace all approaches up till then)
- dataangel 5y ago
  Well, but wasn't AlphaFold at least in part trained based on the existing mechanical models? Or was AlphaFold only trained on ground truth imaging of the folding process?
  - PaulDavisThe1st 5y ago
    It was trained on experimentally-determined physical structures. Not predicted models based on various folding engines.
- 5y ago

eru 5y ago

Not sure why you attack economics? It works really well to eg predict and understand the impact of tariffs or taxes or deregulation. Yes, economists can't predict recessions in advance. And economics also tells you exactly why: the efficient market hypothesis tells you that if people knew that stock prices were going down next year, prices would already react to that knowledge today.

echopurity 5y ago
netcan 5y ago
It's not an attack.
- eru 5y ago
  OK! I might have misread your comment.
rkk3 5y ago
> Yes, economists can't predict recessions in advance Aren't they actually pretty good at predicting recessions? Thought the joke is that they predicted 12 of the last 7.
convolvatron 5y ago
pretty sure the parent is doing the exact opposite. other forms of inquiry aren't valueless because they aren't falsifiable.
- netcan 5y ago
  cheers. Dead on.

snowwrestler 5y ago

You can create a complex system to make predictions about complex systems. But without underlying theory, you can’t put bounds on its accuracy or utility. Such a system will work right up until it doesn’t, with no way to predict in advance where that point is. We see this with people all the time. Jane thinks she knows how Joe will react to X but Joe does something unexpected. A system that makes accurate predictions an unpredictable percentage of the time is not that useful. But it might be great for fooling ourselves or others.

kybernetikos 5y ago
Jane being able to predict how Joe will react most of the time turns out to be super useful in real life. Sure it fails sometimes, but nobody would voluntarily give up that ability.

deleted 5y ago

[deleted]