4 ms·

I realize I've been repeating shibboleths from my postgrad without full understanding. A problem with epicycles is they are harder to falsify. If the model doe

I realize I've been repeating shibboleths from my postgrad without full understanding. A problem with epicycles is they are harder to falsify. If the model doesn't match new observations, just adjust it, or add another epicycle. In contrast, Newtonian gravity can hardly be tweaked at all. So when Mercury's orbit was slightly off, they knew something was wrong. I'm not quite clear on how I feel about this. Geocentricity is a theory, in broad terms. It seems disrespectful to say adding epicycles makes it "not a theory". As you say, there was a theory that the planets actually moved in epicyclic motion. It wasn't just calculation to them. (I want to stress that the idea of epicycles, the mechanical craftsmanship, and actual prediction of the planets are all amazing genius.) Yet, having more parameters than data means the model doesn't explain in simpler terms, only restates. In this sense, it's "not a theory" (by Occam's razor). It seems enough epicycles can model anything: (3Blue1Brown Fourier Series) https://youtube.com/watch?v=bL0LV0Huj1s https://youtube.com/watch?v=bL0LV0Huj1s OTOH the epicycles did predict planetary motion, so they did capture some regularity... not sure what to think. RE Chomsky: You can see it's like epicycles: with enough parameters, an LLM is like a numerical method for curve fitting, that doesn't explain the data (any more than a fourier transform does). Curiously, they do seem to predict very accurately... yet also generalize strangely ("hallucinate"). What to think? But it seems it's got to help! Even if only as a device, like a telescope. Also, from this interview with Terry Sejnowski (https://youtube.com/watch?v=XKC-4Tosdd8 https://youtube.com/watch?v=XKC-4Tosdd8 3 hours!) there's instances where a technique was developed for a problem with Neural Nets, and an equivalent was found in the brain. He also gives a Chomsky-like view: if you duplicated the human brain and it worked perfectly, you wouldn't have done any science if you didn't understand anything. BTW Chomsky's point E (which I'd never heard of), the last and most minor, was based on Gold's work.

YeGoblynQueenne 3y ago
>> RE Chomsky: You can see it's like epicycles: with enough parameters, an LLM is like a numerical method for curve fitting, that doesn't explain the data (any more than a fourier transform does). Curiously, they do seem to predict very accurately... yet also generalize strangely ("hallucinate"). What to think? Well, that's the fundamental problem of modelling: that for any set of observations there's an arbitrary number of models that fit the data with great accuracy and even predict future observations well; and we don't know which one is the best in the long term. The answer is that we should prefer not predictive models, but explanatory theories, that not only predict future observations but also explain why those observations should be expected to be made. For example, the epicyclical model did not explain anything: it said nothing about why the planets should move on circular orbits with epicycles. Kepler's laws didn't explain anything because they didn't say why the planets should move on ellpitical orbits. Newton's law of universal gravitation explained it all in one stroke: because gravity. And that's why we consider Newton the greatest scientist of his era, not Kepler, not Coppernicus, not Gallileo, but Newton, because he explained the world and didn't just describe it. Ultimately the advantage is, like you say, that when an explanatory theory fails, we can better know why. When a predictive model fails, we have no clue. >> BTW Chomsky's point E (which I'd never heard of), the last and most minor, was based on Gold's work. Gold's negative learnability result was a huge upheaval that led directly to the current paradigm of machine learning. Chomsky used it to support his argument about the poverty of the stimulous but linguistics was only one of the two fields that Gold's result turned upside down. And it was a negative result. As I say in another comment, science gives you the tools to know when you're wrong and that's how progress is made, when we find out where we were wrong before. With epicycles, it took almost two thousand years before we figured out where the model was wrong. Let's hope that it doesn't take that long with LLMs and neural nets also, because I doubt we have another couple thousand years to spare on a wild goose chase. >> (I want to stress that the idea of epicycles, the mechanical craftsmanship, and actual prediction of the planets are all amazing genius.) The epicyclical model persisted for so long because it was so good, and because there was nothing better. It is common for people who don't understand science to look at scientists of the past with derision and think they weren't even scientists, but for almost two thousand years, astronomers did exactly what a scientist must do: they accepted the best available theory, even if many of them hated it with a burning passion (and they did!). If it wasn't for the ancients stumbling and fumbling in the dark for millennia, we wouldn't today be enlightened and we owe them every respect.
- foobarqux 3y ago
  > Chomsky used it to support his argument about the poverty of the stimulous but linguistics Do you know where Chomsky refers (directly or indirectly) to Gold? I've been searching for a reference for some time.
  - YeGoblynQueenne 3y ago
    No, I'm sorry. I'm not a linguist so I only know the relation between Gold's result and linguistics second-hand. I'm more interested in it from the point of view of inductive generalisation in machine learning; that's my schtick. Just to make sure I didn't hallucinate all that, I had an admittedly perfunctory search online and I could find this paper: https://proceedings.neurips.cc/paper/2002/file/04ad5632029cbfbed8e136e5f6f7ddfa-Paper.pdf https://proceedings.neurips.cc/paper/2002/file/04ad5632029cb... Whose introduction describes how Gold's result is considered to support the arguments for linguistic nativism from the poverty of the stimulus. Then again, the author doesn't seem to be a linguist himself and he doesn't give any more specific references, so I'm now a little worried; and your question remains un-answered. Have you tried wading through Chomsky's early work on linguistics? I don't have the courage to. The closest I've got to is I have a friend who has read a couple of Chomsky's linguistics books. My friend is making a living as an astrologist now so maybe that's a bit of a warning there :P
  - hyperthesis 3y ago
    (not who you asked) I thought this would be in the linked transcript, but it's not. Norvig must be getting it from elsewhere (maybe in the 404ed video?), but it seems like misrepresentation.
- sdenton4 3y ago
  The gravity model is similar though: we posit a force that pulls things together, but we don't know /why/ that force seems to exist, no more than the ancients knew /why/ the planets seemed to move in smaller circles along their circular paths. We're really not /that/ enlightened, after all.
  - 3y ago
foobarqux 3y ago
> BTW Chomsky's point E (which I'd never heard of), the last and most minor, was based on Gold's work. Do you know where Chomsky says this exactly? I've been looking for how/if his argument is the same as Gold's.

YeGoblynQueenne 3y ago

>> RE Chomsky: You can see it's like epicycles: with enough parameters, an LLM is like a numerical method for curve fitting, that doesn't explain the data (any more than a fourier transform does). Curiously, they do seem to predict very accurately... yet also generalize strangely ("hallucinate"). What to think? Well, that's the fundamental problem of modelling: that for any set of observations there's an arbitrary number of models that fit the data with great accuracy and even predict future observations well; and we don't know which one is the best in the long term. The answer is that we should prefer not predictive models, but explanatory theories, that not only predict future observations but also explain why those observations should be expected to be made. For example, the epicyclical model did not explain anything: it said nothing about why the planets should move on circular orbits with epicycles. Kepler's laws didn't explain anything because they didn't say why the planets should move on ellpitical orbits. Newton's law of universal gravitation explained it all in one stroke: because gravity. And that's why we consider Newton the greatest scientist of his era, not Kepler, not Coppernicus, not Gallileo, but Newton, because he explained the world and didn't just describe it. Ultimately the advantage is, like you say, that when an explanatory theory fails, we can better know why. When a predictive model fails, we have no clue. >> BTW Chomsky's point E (which I'd never heard of), the last and most minor, was based on Gold's work. Gold's negative learnability result was a huge upheaval that led directly to the current paradigm of machine learning. Chomsky used it to support his argument about the poverty of the stimulous but linguistics was only one of the two fields that Gold's result turned upside down. And it was a negative result. As I say in another comment, science gives you the tools to know when you're wrong and that's how progress is made, when we find out where we were wrong before. With epicycles, it took almost two thousand years before we figured out where the model was wrong. Let's hope that it doesn't take that long with LLMs and neural nets also, because I doubt we have another couple thousand years to spare on a wild goose chase. >> (I want to stress that the idea of epicycles, the mechanical craftsmanship, and actual prediction of the planets are all amazing genius.) The epicyclical model persisted for so long because it was so good, and because there was nothing better. It is common for people who don't understand science to look at scientists of the past with derision and think they weren't even scientists, but for almost two thousand years, astronomers did exactly what a scientist must do: they accepted the best available theory, even if many of them hated it with a burning passion (and they did!). If it wasn't for the ancients stumbling and fumbling in the dark for millennia, we wouldn't today be enlightened and we owe them every respect.

foobarqux 3y ago
> Chomsky used it to support his argument about the poverty of the stimulous but linguistics Do you know where Chomsky refers (directly or indirectly) to Gold? I've been searching for a reference for some time.
- YeGoblynQueenne 3y ago
  No, I'm sorry. I'm not a linguist so I only know the relation between Gold's result and linguistics second-hand. I'm more interested in it from the point of view of inductive generalisation in machine learning; that's my schtick. Just to make sure I didn't hallucinate all that, I had an admittedly perfunctory search online and I could find this paper: https://proceedings.neurips.cc/paper/2002/file/04ad5632029cbfbed8e136e5f6f7ddfa-Paper.pdf https://proceedings.neurips.cc/paper/2002/file/04ad5632029cb... Whose introduction describes how Gold's result is considered to support the arguments for linguistic nativism from the poverty of the stimulus. Then again, the author doesn't seem to be a linguist himself and he doesn't give any more specific references, so I'm now a little worried; and your question remains un-answered. Have you tried wading through Chomsky's early work on linguistics? I don't have the courage to. The closest I've got to is I have a friend who has read a couple of Chomsky's linguistics books. My friend is making a living as an astrologist now so maybe that's a bit of a warning there :P
- hyperthesis 3y ago
  (not who you asked) I thought this would be in the linked transcript, but it's not. Norvig must be getting it from elsewhere (maybe in the 404ed video?), but it seems like misrepresentation.
sdenton4 3y ago
The gravity model is similar though: we posit a force that pulls things together, but we don't know /why/ that force seems to exist, no more than the ancients knew /why/ the planets seemed to move in smaller circles along their circular paths. We're really not /that/ enlightened, after all.
- 3y ago

foobarqux 3y ago

> BTW Chomsky's point E (which I'd never heard of), the last and most minor, was based on Gold's work. Do you know where Chomsky says this exactly? I've been looking for how/if his argument is the same as Gold's.