5 ms·

this researcher seems to take it for granted that there is something fundamental/sacrosanct about the "scientific method", but there isn't; it is a learning alg

by hc 17y ago

this researcher seems to take it for granted that there is something fundamental/sacrosanct about the "scientific method", but there isn't; it is a learning algorithm whose role is being diminished by new, more powerful learning algorithms.

RiderOfGiraffes 17y ago
I'm not sure I understand you, and I'd appreciate a more complete analysis. In relation to this article, it says that when you build a model that fits the data, it's not enough. It's a further requirement that the model makes testable predications that turn out to be correct. Models that make accurate predications are valuable. Models that aren't then tested by checking their predictions are, by definition, untested. That, to my mind, makes them potentially dangerous. I'd be interested to know why you describe making falsifiable models is just "a learning algorithm." Thanks.
- hc 17y ago
  > In relation to this article, it says that when you build a model that fits the data, it's not enough. It's a further requirement that the model makes testable predications that turn out to be correct. Models that make accurate predications are valuable. Models that aren't then tested by checking their predictions are, by definition, untested. the problem with the article is that it lumps in many 'models' with wholly untested ones undeservedly, saying And no, that does not mean simply training your model on half of your data set and showing that you can effectively explain the other half of your data. That is hypocritical because this is more or less precisely what scientists, following the "scientific method", are doing, albeit many times slower than a computer with a parametric model. > I'd be interested to know why you describe making falsifiable models is just "a learning algorithm." what happens in some piece of the world over some duration of time can be represented as a function from that piece of the world's initial state to its final state. in the sciences, we try to estimate such functions. the scientific method is a very simple algorithm for doing so: generate a hypothesis, compute N "testable predictions" f(x_i)\approx y_i, and then accept your hypothesis if they all turn out to be correct.
aheilbut 17y ago
There is something very fundamental about the scientific method. Science relies on experiments that ground it in physical reality, which does still exist.
jacquesm 17y ago
> new, more powerful learning algorithms. Could you enumerate those ?
- hc 17y ago
  here is a course i have been working through that describes some learning algorithms: http://courses.csail.mit.edu/6.867/lectures.html http://courses.csail.mit.edu/6.867/lectures.html i also like this book: http://www.amazon.com/All-Statistics-Statistical-Inference-Springer/dp/0387402721 http://www.amazon.com/All-Statistics-Statistical-Inference-S...
  - jacquesm 17y ago
    Learning algorithms and statistical inference do not in any way supersede or invalidate the scientific method, they are products of the scientific method. The are very interesting though, that's for sure.
    - hc 17y ago
      the products of a thing often supersede it. in this case, the development of inference is certainly a consequence of the development of the scientific method. but the validity of the latter is a mathematical consequence of the validity of the former. bayesian inference is more fundamental. another good book that informs my personal views on this matter: http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712 http://www.amazon.com/Probability-Theory-Logic-Science-Vol/d...
caffeine 17y ago
That's actually not true. There is something fundamental about the scientific method. Namely, it's the learning algorithm; the only one that actually actually works (so far). Everything else is just variations on or approximations of this algorithm. When you build a model and then do science (i.e. Hypothesis/Test/Explain loop) on the model instead of doing science on Nature, you learn a lot about your model. But you must be honest and admit that you have learned nothing about Nature. What you might have done is come up with hypotheses. But it's not enough. It's not science yet - it's just a complicated hypothesis. It doesn't count until you test it on Nature. The point (and it's an obvious one, well-understood by most complexity researchers) is that your models need to enable you to falsify them by doing real, repeatable experiments on Nature, not on models. There's no point building a model of a complex system if the only experiment that can falsify it is so complex that it can't be done. A lot of time is wasted this way - in many ways it's the nature of the beast (it's called "complexity" for a reason). Nonetheless, this is frustrating to scientists who think the resources spent on learning facts about models might be better spent on learning facts about Nature - the more radical likely being willing to trade all the modelling (and modelers?) for even a single properly verified fact.
- hc 17y ago
  when you train a computational model on half of your dataset and test it against the other half (which the author claims is Not Sufficiently Scientific), how is that qualitatively different from training it on all your data, and then "testing it on Nature" by gathering some more and checking your results? i am reminded of searle's chinese room argument. > this is frustrating to scientists who think the resources spent on learning facts about models might be better spent on learning facts about Nature - the more radical likely being willing to trade all the modelling (and modelers?) for even a single properly verified fact. there is no getting rid of models. neither science nor any other human activity i am aware of is capable of verifying facts about the physical world.
  - jacquesm 17y ago
    A learning algorithm creates a specialist filter, usually in classification problems, that can discriminate between the various classes. That doesn't mean that because you didn't manually tune each and every weight in the classifier that its behaviour is not solidly governed by scientific principles. Science and the application of the scientific method are what got us there in the first place, and has given us the keys to understanding these systems. Genetic Algorithms are a another area where you could easily be tempted to think that you are not doing science, the same thing happens there. Science is the key to understanding. The scientific method is the ultimate arbiter between what we know to be true, and what we can not prove to be either false, fantasy or outright lie. I feel that you think that somehow these computational models somehow supplant the scientific method, but you are forgetting one crucial bit here, those models themselves will need to be understood, and only the scientific method will allow you to do so.
deleted 17y ago
[deleted]
zeynel1 17y ago
@hc: This is so true. In my opinion "scientific method" is actually undefined. Just checked Wikipedia to confirm that "scientific method" itself has become a bloated academic field with countless versions; it is not used by practicing researchers. http://en.wikipedia.org/wiki/Scientific_method http://en.wikipedia.org/wiki/Scientific_method
- jacquesm 17y ago
  The scientific method is alive and well, according to that same article you linked above these are the steps in 'doing science' following 'a scientific method' (of which there are many, but they all share the same basic characteristics): 1. Define the question 2. Gather information and resources (observe) 3. Form hypothesis 4. Perform experiment and collect data 5. Analyze data 6. Interpret data and draw conclusions that serve as a starting point for new hypothesis 7. Publish results 8. Retest (frequently done by other scientists) So, even though because of the majority of fields and scope of problems we can not use a single methodology to give us all our knowledge (that would be so convienient) we have come up with a series of steps that if you adhere to them will give you meaningful results.
  - zeynel1 17y ago
    Yes this is my point. Is there a "The Hacking Method?" I believe not. There may be lists such as "7 good habits of best hackers" but not a Biblical commandment defining a method to be the true computer programming. The idea of an absolute sacrosanct Scientific Method is contrary to what scientific research is about. It defines science to be an academic method. So, assume that I have my personal system of doing research. I don't define the problem. I don't form formal hypotheses; I don't perform experiments (formal academic exercises) instead I use trial and error; I take frequent stratetegic naps to develop my ideas; I analyze the idea any way I like; I don't publish my results; instead I note my results meticulously on my lab books. I don't care anything about academic rules and ignore academics and their science fetish.... instead I make discoveries. I ignore the paperwork, and career considerations. . . I make discoveries. You recognize Edison's method. Are you saying that what I am doing is not science because I did not follow your step by step "scientific method?" The scientific method is an attempt to brand academic physics as science and the rest as stamp collecting. This marketing gimmick actually has been working. People associate academic physics with science.

by hc 17y ago

RiderOfGiraffes 17y ago

I'm not sure I understand you, and I'd appreciate a more complete analysis. In relation to this article, it says that when you build a model that fits the data, it's not enough. It's a further requirement that the model makes testable predications that turn out to be correct. Models that make accurate predications are valuable. Models that aren't then tested by checking their predictions are, by definition, untested. That, to my mind, makes them potentially dangerous. I'd be interested to know why you describe making falsifiable models is just "a learning algorithm." Thanks.

hc 17y ago
> In relation to this article, it says that when you build a model that fits the data, it's not enough. It's a further requirement that the model makes testable predications that turn out to be correct. Models that make accurate predications are valuable. Models that aren't then tested by checking their predictions are, by definition, untested. the problem with the article is that it lumps in many 'models' with wholly untested ones undeservedly, saying And no, that does not mean simply training your model on half of your data set and showing that you can effectively explain the other half of your data. That is hypocritical because this is more or less precisely what scientists, following the "scientific method", are doing, albeit many times slower than a computer with a parametric model. > I'd be interested to know why you describe making falsifiable models is just "a learning algorithm." what happens in some piece of the world over some duration of time can be represented as a function from that piece of the world's initial state to its final state. in the sciences, we try to estimate such functions. the scientific method is a very simple algorithm for doing so: generate a hypothesis, compute N "testable predictions" f(x_i)\approx y_i, and then accept your hypothesis if they all turn out to be correct.

aheilbut 17y ago

There is something very fundamental about the scientific method. Science relies on experiments that ground it in physical reality, which does still exist.

jacquesm 17y ago

> new, more powerful learning algorithms. Could you enumerate those ?

hc 17y ago
here is a course i have been working through that describes some learning algorithms: http://courses.csail.mit.edu/6.867/lectures.html http://courses.csail.mit.edu/6.867/lectures.html i also like this book: http://www.amazon.com/All-Statistics-Statistical-Inference-Springer/dp/0387402721 http://www.amazon.com/All-Statistics-Statistical-Inference-S...
- jacquesm 17y ago
  Learning algorithms and statistical inference do not in any way supersede or invalidate the scientific method, they are products of the scientific method. The are very interesting though, that's for sure.
  - hc 17y ago
    the products of a thing often supersede it. in this case, the development of inference is certainly a consequence of the development of the scientific method. but the validity of the latter is a mathematical consequence of the validity of the former. bayesian inference is more fundamental. another good book that informs my personal views on this matter: http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712 http://www.amazon.com/Probability-Theory-Logic-Science-Vol/d...

caffeine 17y ago

That's actually not true. There is something fundamental about the scientific method. Namely, it's the learning algorithm; the only one that actually actually works (so far). Everything else is just variations on or approximations of this algorithm. When you build a model and then do science (i.e. Hypothesis/Test/Explain loop) on the model instead of doing science on Nature, you learn a lot about your model. But you must be honest and admit that you have learned nothing about Nature. What you might have done is come up with hypotheses. But it's not enough. It's not science yet - it's just a complicated hypothesis. It doesn't count until you test it on Nature. The point (and it's an obvious one, well-understood by most complexity researchers) is that your models need to enable you to falsify them by doing real, repeatable experiments on Nature, not on models. There's no point building a model of a complex system if the only experiment that can falsify it is so complex that it can't be done. A lot of time is wasted this way - in many ways it's the nature of the beast (it's called "complexity" for a reason). Nonetheless, this is frustrating to scientists who think the resources spent on learning facts about models might be better spent on learning facts about Nature - the more radical likely being willing to trade all the modelling (and modelers?) for even a single properly verified fact.

hc 17y ago
when you train a computational model on half of your dataset and test it against the other half (which the author claims is Not Sufficiently Scientific), how is that qualitatively different from training it on all your data, and then "testing it on Nature" by gathering some more and checking your results? i am reminded of searle's chinese room argument. > this is frustrating to scientists who think the resources spent on learning facts about models might be better spent on learning facts about Nature - the more radical likely being willing to trade all the modelling (and modelers?) for even a single properly verified fact. there is no getting rid of models. neither science nor any other human activity i am aware of is capable of verifying facts about the physical world.
- jacquesm 17y ago
  A learning algorithm creates a specialist filter, usually in classification problems, that can discriminate between the various classes. That doesn't mean that because you didn't manually tune each and every weight in the classifier that its behaviour is not solidly governed by scientific principles. Science and the application of the scientific method are what got us there in the first place, and has given us the keys to understanding these systems. Genetic Algorithms are a another area where you could easily be tempted to think that you are not doing science, the same thing happens there. Science is the key to understanding. The scientific method is the ultimate arbiter between what we know to be true, and what we can not prove to be either false, fantasy or outright lie. I feel that you think that somehow these computational models somehow supplant the scientific method, but you are forgetting one crucial bit here, those models themselves will need to be understood, and only the scientific method will allow you to do so.

deleted 17y ago

[deleted]

zeynel1 17y ago

@hc: This is so true. In my opinion "scientific method" is actually undefined. Just checked Wikipedia to confirm that "scientific method" itself has become a bloated academic field with countless versions; it is not used by practicing researchers. http://en.wikipedia.org/wiki/Scientific_method http://en.wikipedia.org/wiki/Scientific_method

jacquesm 17y ago
The scientific method is alive and well, according to that same article you linked above these are the steps in 'doing science' following 'a scientific method' (of which there are many, but they all share the same basic characteristics): 1. Define the question 2. Gather information and resources (observe) 3. Form hypothesis 4. Perform experiment and collect data 5. Analyze data 6. Interpret data and draw conclusions that serve as a starting point for new hypothesis 7. Publish results 8. Retest (frequently done by other scientists) So, even though because of the majority of fields and scope of problems we can not use a single methodology to give us all our knowledge (that would be so convienient) we have come up with a series of steps that if you adhere to them will give you meaningful results.
- zeynel1 17y ago
  Yes this is my point. Is there a "The Hacking Method?" I believe not. There may be lists such as "7 good habits of best hackers" but not a Biblical commandment defining a method to be the true computer programming. The idea of an absolute sacrosanct Scientific Method is contrary to what scientific research is about. It defines science to be an academic method. So, assume that I have my personal system of doing research. I don't define the problem. I don't form formal hypotheses; I don't perform experiments (formal academic exercises) instead I use trial and error; I take frequent stratetegic naps to develop my ideas; I analyze the idea any way I like; I don't publish my results; instead I note my results meticulously on my lab books. I don't care anything about academic rules and ignore academics and their science fetish.... instead I make discoveries. I ignore the paperwork, and career considerations. . . I make discoveries. You recognize Edison's method. Are you saying that what I am doing is not science because I did not follow your step by step "scientific method?" The scientific method is an attempt to brand academic physics as science and the rest as stamp collecting. This marketing gimmick actually has been working. People associate academic physics with science.