3 ms·

Are ”noisy” inputs here at all related to ones where their Kolmogorov complexity is their encoding length? I don’t know how much I buy the idea that intelligen

• jhanschoo 11mo ago
  If we accept the functional framing (as being able to give a suitable suggestion conditioned on input), then it seems to me that parsimony is the only sensible general framing; every deviation from that is something that is specific to an application or another and can be modeled by a transformation of the input space/output space. > There are probably cases where a small TM explains a very large but finite set of observations, but if a few new ones are added the parsimonious explanation becomes much longer and looks much different from the previous one. Indeed, to use an analogy, if you have 99 points that can be described perfectly by a linear function except for one outlier, then clearly your input isn't as clear-cut as might have been originally assumed. On the other hand, you may be in a different setting where you have noisy sensor inputs and you expect some noise, and are looking for a regression that tolerates some noise. In such a situation, only when the stars align perfectly would your input data be perfectly described by a linear function, and we just have to accept that a broken watch is perfectly right twice a day whereas a working one is almost always only approximately right, but all the time.
  • deleted 11mo ago
    [deleted]
  • mxkopy 11mo ago
    Ah, what I was hoping to get at is that true intelligence might not have these big gaps between explanation-lengths that parsimonious TMs do. And there’s also the question of deduction; having a few redundant “theorems” on hand might make deductive inferences more efficient whereas parsimony would elide them. All this to say I hope there are some gaps in our theoretical understanding of true AI, otherwise I wouldn’t be able to make a living filling them in
    • jhanschoo 11mo ago
      Ah I now think I know what you were thinking of when you were talking about "noisy" in terms of K-parsimony, you were thinking of maximally random input strings. > is that true intelligence might not have these big gaps between explanation-lengths that parsimonious TMs do. I don't know the field and literature well enough to know if this is the case, is there a published result you can point me to? > And there’s also the question of deduction; having a few redundant “theorems” on hand might make deductive inferences more efficient whereas parsimony would elide them. Especially with the words "redundant", "deductive", and "efficient", it sounds to me that you have in mind something like CDCL SAT solvers learning redundant conflict clauses that help prune the search space. In respect to this recall that the AIXI definition/Solomonoff induction definition is noncomputable and so doesn't have a notion of efficiency. Indeed, some optimally parsimonious TMs for some inputs are not going to meet fixed resource bounds on part of the input. Intuitively if you are concerned about a finite part of the input space, you can just tack them on to the definition of the TM to obtain a TM that has good efficiency on that finite space, at the cost of definitional parsimony. Possibly something in-between for particular infinite spaces exist (dovetailing with a more complex TM with better runtime characteristics that agrees on that space?) and I wonder if there might very well be an efficient frontier of parsimony against say time complexity.
      • mxkopy 11mo ago
        Right, I’m not the most well read on this stuff either, so I’m wondering now if existing architectures operate on this > efficient frontier of parsimony against say time complexity. As you mentioned before regularization approximates parsimony, could it be that what’s gained from this loss of precision wrt parsimony are runtime guarantees (since now we’re mostly talking about constant depth circuit-esque DL architectures)? Or is the jump to continuous spaces more relevant? Are these the same? I’ll have to read up more to see