3 ms·

I think you have fallen into the trap of mistaking interpolation for generalization. Working with these models every day, it's clear that they can certainly in

I think you have fallen into the trap of mistaking interpolation for generalization. Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise. That's not to say that interpolation in a rich latent space of text isn't very useful. But it's not the same level of abstraction that comes from true generalization in this space.

riwsky 3y ago
I think we’ve all fallen into the trap of mistaking pithiness for evidence
famouswaffles 3y ago
>it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise Failing to generalize on whatever you have in mind is not evidence that the models are incapable of generalization. If you really think so, just be prepared to write off a good chunk of humans as well lol. This is generalization. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ Just seems to me like you've taken examples of generalization and invented an alternate meaning just so you can't admit it generalizes. "Oh but you see this example of answering an unseen question is "insert meaningless distinction" so it doesn't really count"
johndough 3y ago
With a sufficiently large dataset, every problem becomes interpolation.
nl 3y ago
Given a multi-dimensional latent space with enough dimensions it's hard to imagine cases of generalization that aren't interpolation between points in latent space (given enough data). The one possible exception is logical inference, and this problem seems tractable with tool use or programming.
- Y_Y 3y ago
  I object on geometric grounds. You can't interpolate outside the convex hull, if your have an outlier in any dimension then you're going to need to extrapolate, that seems to me like a reasonable way to ask for generalisation that isn't interpolation.

riwsky 3y ago

I think we’ve all fallen into the trap of mistaking pithiness for evidence

famouswaffles 3y ago

>it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise Failing to generalize on whatever you have in mind is not evidence that the models are incapable of generalization. If you really think so, just be prepared to write off a good chunk of humans as well lol. This is generalization. https://general-pattern-machines.github.io/ https://general-pattern-machines.github.io/ Just seems to me like you've taken examples of generalization and invented an alternate meaning just so you can't admit it generalizes. "Oh but you see this example of answering an unseen question is "insert meaningless distinction" so it doesn't really count"

johndough 3y ago

With a sufficiently large dataset, every problem becomes interpolation.

nl 3y ago

Given a multi-dimensional latent space with enough dimensions it's hard to imagine cases of generalization that aren't interpolation between points in latent space (given enough data). The one possible exception is logical inference, and this problem seems tractable with tool use or programming.

Y_Y 3y ago
I object on geometric grounds. You can't interpolate outside the convex hull, if your have an outlier in any dimension then you're going to need to extrapolate, that seems to me like a reasonable way to ask for generalisation that isn't interpolation.