7 ms·

The level of analysis deep learning systems get is considerably out-of-proportion, and speaks to a superstitious view of how ML works. Namely, that it uncovers

The level of analysis deep learning systems get is considerably out-of-proportion, and speaks to a superstitious view of how ML works. Namely, that it uncovers latent representational structure not present in the data. (Incidentally, its clear this book doesnt make this mistake, already identifying that NN is basically kNN). Rather ML algorithms are just rememberings of data coupled with various degrees of compression, with kNN having zero compression -- just a straight weights=data; and NNs having fairly significant levels, where weights=ensemble(compressions(data)). We should therefore regard the ML step as incredibly trivial, it is just a clever process of averaging over the data given to it. The whole "magic" of ML, such as it is, is *only* in the data. And this is where the word "data" hamstrings our ability to see properly. Everything *isnt* data; and "data" isnt some source of information. The world exists, and "data" is just what we call any measurement of any part of it by any means. "Data" is only *relevant* to the probelm we're trying to solve if we do incredible amounts of experimental work to carve-the-world along its joints, ie., to have the right concepts; and incredible amounts of work to measure along its joints, ie., to have the right units. *And then* to eliminate all the coincidences and irrelevances. *And then* to provide that to a machine, which at this point, does basically nothing but automate our effort. Almost all data it is possible to collect is useless, indeed, an infinite amounts is useless. The magic of ML is a sleight-of-hand trick -- we dont really need to know how its averaging of our data does anything useful -- it almost never does. Rather, it is our "experimental design" which produces the usefulness of the system. ML algorithms are just interpolations and averages through data prepared to produce useful averages by (literally millenia) of human ingenuity. It takes actual intelligence to do this because the world isnt data, and almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to "be in the world" to resolve; and resolution is a dynamic process which you "have to be here for".

nkozyra 5y ago
It sounds like the unique attribute you're describing in humans is nuance and contextual understanding, but we developed those as a way of doing something a neutral network does inherently by means of error adjustment. I don't think there's ever an expectation that the data is "truth" but that we have to contextualize it and adjust it. Humans adapted to those through different mechanisms than NN but the goal is still fundamentally the same: to predict something with as much accuracy as possible despite the infinite noise of the data that is the world.
- mjburgess 5y ago
  Alas the goal isnt the same, and here "predict" is the culprit. What ML systems do is offer statistical estimates, which is one very narrow meaning of "predict". When I "predict" what will happen when a glass falls, I'm not giving you any statistical estimates. I'm running a mental simulation of the world in its relevant parts and using this to predict what will happen. The heart of basically almost all knowledge of the world is actually counterfactual (ie., requiring simulation), it's always "if this happens (in the simulation), this will happen (in the simulation)". To acquire this ability to predict one has to understand, ie., to have the right concepts, reason with them and so on. When a person "has the concept `7`" they can reason thus: "6 eggs would be fewer", "7 is an odd number", "7 is a whole quantity", "with 7 less of 10, i'd have 3" etc. Ie., `7` is available to play a fundamental role in simulating scenarios and reasoning with them. When a NN produces a mapping from {Images} -> {Labels} such that the squiggle `7` produces the label `"7"` it doesnt do so via prediction in the meaningful sense of the world. Rather it has found a means of averaging sqiggles, and compares one squiggle to its averages and reports what the historical similar squiggles were labelled. This is a game which is only useful because we, who can reason with `7` profoundly about the world (etc.), have produced datasets which express our knowldge of the world in ways that dont require it. Ie., we present to the machine something whose average is useful. The machine is not trying to, nor does it, nor can it, predict. It has no knoweldge at all. It is just a correspondance table we have provided.
  - daveed 5y ago
    I hear what you're saying, but I would not state so confidently what prediction "is" and what concepts "are", in the human model.
    - mjburgess 5y ago
      I think this conversation gets away from itself very quickly if you run with generic terms: data, prediction, and so on. People think that NNs/ML/etc. works, and now the relevant questions are philosophical: if it works, how. But it doesnt work. In very silly trivial ways. It's only taking averages over historical data. And all it can do is that. As soon as you are alive to what thinking about things requires (ie., a live embeddedness into an environment, imagining scenarios, etc.) it's trivial to expose the magic show. It isnt philosophical, it's quite literally just broken and doesnt work. I cannot ask GPT, "do you like what i'm wearing?", or "what would it take to change your mind on drug legalisation?" -- and so on. Indeed, I can't ask it, "if I have seven eggs, should I make a cake with 6 and keep the 7th -- or should I just use 7?". I can't ask it anything which actually requires it to reason with concepts, because it doesnt have any; and I cant ask it about anything around now because it is just summarising what text has already been produced. The whole AI hype industry has a vested interest in the philosophy here, ask its premised on everythign working. Almost nothing works. The idea that GPT presents any philosophical challenges is nearly absurd; about as many as a shredder at a library.
clircle 5y ago
"We have more data than ever, more good data than ever, a lower proportion of data that are good, a lack of strategic thinking about what data are needed to answer questions of interest, sub-optimal analysis of data, and an occasional tendency to do research that should not be done."
youhavetosayit 5y ago
<< almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to "be in the world" to resolve; and resolution is a dynamic process which you "have to be here for". >> Sounds like a magic view of the mind to me. It's suspicious that this is never said about the function of Liver or Kidneys, only about the function of the Brain. Maybe the brain is protecting itself, like if you ask a cow about steaks.
- mjburgess 5y ago
  The brain doesnt do it, the body does. The way we can tell if a stick is bent, or not, in a fluid is to remove the stick. Animals arent magic, theyre just actually present in the world. The converse is true: if the world's structure could be uncovered by data analysis, then it would be something like The Matrix; and the whole history of science would be falsified. It turns out the hardwork of experiment was never required, one could just blindly measure anything in any way, and somehow, knowledge would be produced.
  - lifeisstillgood 5y ago
    So (trying to grok this) in order to "know" I have picked the box up, I can feel its weight, wave my hand in the empty space etc. I am "in the world" and use side channels to build my own reward function for my own eyesight. (presumably this is the first 9+ months of life. if I only had eyesight i could not build a reward function ? Does this mean AI could do this if they had other sensors like weight gauges in the servos?
    - mjburgess 5y ago
      Well the body has properties which enable it to formulate concepts which begin simply as motor techniques. It is unlikely that servos have the right properties, but it's an open question. I strongly suspect the body's ability to organize itself at the cellur-to-organ level over time, ie., to grow, is an essential component of how we develop & deploy novel motor techniques. It is these techniques which we use to resolve the ambiguity inherent within any system of measurement -- we, of course, learn to see -- but i think "learn" here is really a complex form of motor control which ends up structruing our perception and then reflexivily, we build on that to develop more motor techniques. We learn to see in the sense that we first learn to move, then we see better, and we cycle until we're building electron microscopes. Explicit cognition is really just a book-keeping/accounting system to tidy up this process. In modelling only this, we're relying on "doing all the hardwork" for machines. Intelligence isnt what cognition does when it tidy's up your concepts, its the very having of those concepts. Cognition is a big, largely neurotic unself-aware misdirection. It's what academics think is important -- failing to notice they have any body at all. They only got their cherished abstractions by first moving their meat -- and whilst that can be dropped for the most pure and formal of ideas (eg., arithmetic) -- it cannot be dropped for discovery as such.
jah242 5y ago
There might be some truth in what you say for very large image and language models that use supervised learning. It is really hard to see how this 'it's just a lot of good data' view applies to deep reinforcement learning where the model learns multi step policies from raw input data (e.g a camera on a robot) with only a rough high level reward function to guide it. If therefore (as seems to be the case) you can abstract the information humans need to provide to the model/learning system to ever high levels of reward function (and thereby vastly reduce the information provided by humans) then it seems very hard to argue that the model (and the training process) isn't doing to some degree what you describe as: 'incredible amounts of experimental work to carve-the-world along its joints, ie., to have the right concepts; and incredible amounts of work to measure along its joints, ie., to have the right units. And then to eliminate all the coincidences and irrelevances.' For example, imagine a robot learning from scratch to pick objects up based on raw pixel data with only a scalar reward function - where in this process is the human preparing the data so the model only has to average?
- mjburgess 5y ago
  > For example, imagine a robot learning from scratch to pick objects up based on raw pixel data with only a scalar reward function - where in this process is the human preparing the data so the model only has to average? Great -- so do you have an example of such a system? I'd be inclined, initially, to deny that it exists. If your reward function expresses a reward for the goal of "picking up objects with (pixel-space) properties etc.", you're cheating. In this case, the reward function serves the role of the data: ie., prepared by us to work. Indeed, a function is just a dataset -- and the reward function here is being sampled by the system. You'd need to show me a system whose reward function / dataset didnt "contain the solution", in the manner of animals who respond to the world without already having all the information about it. The relevant capacity a system needs to have, in both cases, is being able to take a profoundly ambiguous environment and produce a dataset/reward-fn which "carves along its joints". Ie., which effectively eliminates that ambiguity. When such ambiguity & coincidence is eliminated, there's basically nothing left to do -- it's that basic nothing which we task machines with doing. Ie. running `mean(sample(unambigious relevant well-carved data))`. You'll note its the *properties* of the data which express intelligence & learning.
euphetar 5y ago
As an ML Research Scientist, I have never heard this interpretation. It's a very interesting thought that NN == kNN. It puts some of my lingering intuitions in clear wording. Thank you for this. I think you are close to truth. This would explain why even the largest language models can't generalize beyound the training set. At the same time, I disagree that analysis is out of proportion. It might be some clever averaging, but it does useful and interesting things. Take a look at Google: some clever averaging can get you a long way. It would be great to understand how it works and how can we go beyond it. I do believe we need a paradigm shift, but it does not come out of nothing.
- mjburgess 5y ago
  I am myself trying to get closer to a clear formulation of this problem, which is why I'm writing here. Here's what I have so far, ML systems (eg., NN) remember averages (, compressions) of historical data. They are useful, wrt the problem, iif (1) the problem's target function exists; (2) the data is relevant, unambiguous, well-carved; and if (3) these properties will hold regardless of likely permutations to the problem's framing. Systems are given data with these properties by significant amounts of experimental design, work, and effort by people. Absent these properties, data is useless. Producing data with these properties requires intelligence, and no machine systems exist which can do it. My issue with research into ML on the whole, is that it *assumes* these properties and then explains how the systems work. I understand why this is interesting from a formal perspective... but it fails to note that this situation is almost never how ML is used. There is no function from Image->Animal, ie., biologists arent just idiots who could have just looked at some pixel patterns. Pixel patterns are radically ambigious wrt to `Animal`, and so even an infinite sampling of (Image, Animal) is not enough for ML. ... so what on earth are ML systems doing? This is a bigger research question: to characterise how ML performs when this assumed setup fails. And you know, that research almost doesnt exist. This is an industry led by partisans to its success. What do you think would happen if research actually talked about the dynamics of ML systems performance when (1) the target doesnt exist; (2) data isnt relevant & unambigious; (3) the problem framing will permute most times its deployed.... Suddenly we'd have an explaination of why 2016 wasnt the year self-driving cars were delivered. And indeed, likewise, of why even 2036 wont be.
vegesm 5y ago
I don't really get the first part of your comment. You say an NN is "just" compression + kNN and does no representational learning. But finding a compression (a transformation in other words) that makes kNN feasible on the data is exactly what people mean when they say it finds a hidden representation. It is a highly non-trivial task: e.g. simple distance in pixel space between images would get you useless results.
- mjburgess 5y ago
  People have the notion that a latent representation in the animal sense, ie., a concept -- is the same thing as your "representation" in the NN sense. That's not the case. You're right that if I find a predictive compressions of faces, say F1...n then they arent literal "rememberings". And they seem to be able to participate in a decision process (eg., classification) which doesnt seem to target pixel patterns. However I think this is kind of an illusion. What `F1..n` are, are ambiguous pixel-space projections of the abstraction which isnt present in this projection. When I have the concept "this type of face" I can reason with it beyond similarity in pixel-space. When we form representations we arent restricted to reasoning with them in only one space (eg., how faces look as pixels). We (perhaps superstitiously) impart to machine "representations" an actual depth which they lack. They are templates derived from the spaces they live in, eg., pixel-space; and have only the properties that space affords (eg., pixel-geometry). Reasoning beyond that space, and those properties, doesnt work. People think it does. This is the illusion. Templates derived from this data, that we provide, function like actual representations because we simplify the world for the machine -- and prepare its environment so that its pixel-space templates are good enough.
  - PaulDavisThe1st 5y ago
    I've been with you so far, but .. > When I have the concept "this type of face" I can reason with it beyond similarity in pixel-space. I think there are at least two possible things that might going on here: 1. we're "trained" on non-pixel data (to use the same framing) and so it seems obvious that we would reason with a concept like "this type of face" in a non-pixel space. 2. the experience of "reasoning with it" is an illusion, and is merely the subjective experience that we have when our brains do stuff with whatever their underlying representation is. That is, we may have no real knowledge of what space our own "face model" is built on, what it represents, what properties it considers.
mjhirn 5y ago
I have been working and thinking about data and ML models for many years and this is probably the most concise and lucid summary of a post NN philosophy I have heard so far. It also resonates with me. Thanks for sharing.
s_Hogg 5y ago
I like what you're saying, but doesn't this just reduce to "garbage in, garbage out"?
- mjburgess 5y ago
  Well if you, like me, see ML as only a garbage shaft, then yes. But a lot of people think of it as a fancy processing plant which can take sewage and make diamonds. Really, it takes carefully prepared gemstones and crushes them into a shiny powder.
zmgsabst 5y ago
This is self-righteous nonsense. DNNs end up encoding parts equivalent to type theories in their operations — extracting an “internal language” (in the TT/CAT) sense which models the data. In particular, convolutions are a form of extracting/deducing types from your input. A full DNN deduces a model of the data in which it can type the incoming data and then reason about relations on those types to decide the challenge. What is it that you think you’re doing, besides forming a model from data in which you reason based on an internal language, which describes the data you’ve seen? Your post comes across as ignorant of eg, the overlap between TT and DNNs, while simultaneously full of woo about how humans operate.
- mjburgess 5y ago
  Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. What I think we're doing is, you know, forming concepts which are not constrained to spaces. I dont even really think our concepts, which are mostly just bundles-of-techniques, live within the space of the target object, nor its measurment space (ie., that of experience). Eg., my "pen" includes being able to grasp a pen (indeed, the relevant motor skills are prior-to and necessary-for the abstract concept formation). What you're talking about is a pixel-space projection of "pen" being able to function like the (animal, body-first) concept "pen". It doesnt. Rather "pen" in pixel space is, sure, an "internal langauge" of a sampling of AllPixelPatterns concerned with pixel-space projections of "pen". This i'd call a "template", and I dont think it has almost anything to do with concepts/representations/etc. What we are doing when we acquire motor techniques which produce "ways of sensing and moving" that eventually could become reified as the abstract "Pixel Pen" is really nothing at all like sampling PixelPenSpace and deriving a tempalte. To see the difference, consider that our concept allows us to resolve ambiguities -- eg., if I think something might be a pen, I can go beyond one space of measurement (eg., sight) -- eg., move the pen, write with it etc. -- and thus return to the target space "Is Pen?" with sufficient confidence. Once such "mere templates" exist, everything we do, isnt required. Indeed, a calculator can take over at that point; put it in a cupboard forever to repeat whatever thin inferences pixel templates admit.
  - zmgsabst 5y ago
axg11 5y ago
NN == kNN is a really interesting thought, but I think it’s the wrong framing. All learning is bound by kNN, including, I would argue, human intelligence. When we compare and contrast things, be it words, objects, or images, the very best we can do is to compare those “units” to all other examples that we have ever observed. There is no such thing as generalising beyond kNN. The real magic in generalising is not how many nearest neighbours you are able to compare against, the magic is in _which_ qualities you use to determine the kNN. Another way to state this is: intelligence is in the distance metric/embedding/compression, not the ability to process all examples. When we say that humans are able to generalise, what we really mean is that we are able to compare and contrast specific subsets of qualities for a given object against all other things objects we have knowledge for. The way we choose and weight those specific qualities is what results in our “intelligence”. So when we say that NN == kNN, I think this is actually a really impressive achievement for neural nets. Neural networks are able to produce kNN level results with orders of magnitude lower compute. Humans too, are able to do this, and if we’re being very thorough, we sometimes fall back to kNN-like processes
- mjburgess 5y ago
  Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. Yes, kNN is the bound if you formulate learning as: there exists an f_true: Pixels->Animals; we have an infinite sampling of (Pixels, Animals); kNN performs at 100%. Therefore kNN is the bound. Whereas I dont think this has much to do with learning. Learning for me is: the function probably doesnt exist, we dont know what its domain would even be, and we have no idea how to sample that space. You know: the history of human civilization, but also: the history of a day of any animal's life. We encounter problems daily whose functions can only be modelled by partial input and output domains, and our job is to find ways of resolving the inherent ambiguities in those situations. Ie., to find what parts of the world are relevant, to find ways of measuring it, and hence to find more data until the ambiguity resolves itself. kNN being a perfect algorithm is a catastrophe for this whole formulation of AI. All it means is that the whole field of machine learning has as its limit what you can do if you already know the answer. You can see from this that it isnt learning at all. The situation is exactly backwards: ML assumes relevant well-carved unambiguous data. The whole problem of learning is precisely to produce it!
canjobear 5y ago
If neural networks do compression, then they are implicitly providing structure not present in the data. Given a dataset there are always many different ways to compress it, each corresponding to some implicit distribution over unseen datapoints. Most of these distributions don’t correspond to real-world generalizations. Yet neural networks do often—not always—choose models for compression that correspond to real generalizations. This is nontrivial.
vladf 5y ago
This comment is making a "slight of hand" itself: how do we define the metric for kNN? That will directly impact kNN performance. Using Euclidean norm over pixels on imagenet will not get you anywhere. By recasting classification into a kNN lens you're basically (in the large data regime) recasting the problem to kernel learning under some underspecified RBF kernel with an unknown "proximity" metric instead of Euclidean. In grad school, someone in my lab actually tried scaling kernel methods directly to imagenet too. I don't think that ended up working naively (an interesting neural variant worked for CIFAR10, though [1]). If all NNs are doing is efficiently learning an appropriate kNN metric (or, more idiomatically and generally put, learning kernel parameters for some implicit neural kernel), that's still really powerful and all we've done is just renamed "learning representations." [1]: https://arxiv.org/abs/2003.02237 https://arxiv.org/abs/2003.02237
dotnet00 5y ago
I think NN=kNN is reductive to the point of near uselessness as a statement. You can treat human object recognition as a sort of kNN as well in that we 'detect' a large number of features, the grouping of which ultimately allows us to recognize something. But of course that doesn't actually tell us much about how we recognize things, since that explanation just changes the question to "how do we detect useful features?". Similarly, I think most ML people recognize that NN classifiers are just learning to look for groups of features, yet that doesn't really count as an analysis because what we want to understand is why it arrives at the features it does. >The magic of ML is a sleight-of-hand trick -- we dont really need to know how its averaging of our data does anything useful -- it almost never does. I strongly disagree, we do need to know how the 'averaging' does what it does so we can improve our models further. A lot of big steps in NN architectures have come in part from applying theories and insights regarding the properties of the training process, so a deeper understanding of the process may allow us to make better models, ones that have better accuracy as well as ones that train faster with less data. >Rather, it is our "experimental design" which produces the usefulness of the system. ML algorithms are just interpolations and averages through data prepared to produce useful averages by (literally millenia) of human ingenuity. >It takes actual intelligence to do this because the world isnt data, and almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to "be in the world" to resolve; and resolution is a dynamic process which you "have to be here for". I think most of the industry agrees that current ML models are sort of like being able to make the icing but not the cake (i.e AGI), so I'm not really sure what you're trying to say. Of course the models need a general intelligence to filter general data down to a specific problem and set things up for the model to learn?
djoldman 5y ago
My hot take is that this correct in spirit. However, it just moves the conceptual line between data/feature engineering and modeling so as to make the data/feature engineering encompass nearly all of the process. By this definition of ML, one might just say that clever ML is a really clever way to engineer useful features, after which we "merely" average them. It feels like semantics.
- mjburgess 5y ago
  From a certain POV the structure of an animal's body, as for example, it learns to ride a bike can be given this kind of formulation. Namely, that the muscle tissue and the motor cortext express a latent representation of riding which is formed under the animal's self-regulative goals of "stability" and "getting to some location" etc. From this POV, one can model any aspect of reality with this sort of language. I used to be annoyed by this, but less so now. Annoyed because it's pretty hopelessly useless describe any complex physical system this way. It only makes sense if your relevant "feature spaces" are discrete and computationally tractable... i dont think our muscles are good candidates. (Nor likewise, is almost anything). Nevertheless, you are right that "ML" on a domain whose dimensions are infinite, and in which slices through those dimensions are infinite, and whose models are chaotic -- then ML with an infinity of time to run would produce an animal. And indeed, an intelligent system. One has to remember here that no one is doing this. What theyre doing is taking a highly regulated non-chatoic, tractable, measured and largely unambigious slice through these spaces --- sampling them --- and taking an average. Who gave you the knife to take this slice? Was it your cognition, was it the world? No, neither, it was your body -- largely; and its ability to regulate your imagination; and to guide action. Formulating reality as-if its patterns can be determined statistically leads to absurd levels of non-computability. Every possible partitioning of the world, in every possible arrangment, moving in every possible way -- what discrete model of this is there? There isn't one. And this isnt semantic. The challenge of building an intelligent system is a (bio-)engineering one: we require a material which grows intelligently, whose internal chaos is useful. Not a dumb little calculator which comes in to average our efforts after the fact.
  - neatze 5y ago
sdenton4 5y ago
"Namely, that it uncovers latent representational structure not present in the data." Who thinks this? You're talking to straw-men. The latent structure is absolutely present in the data, but a series of transforms makes the structure more readily available. (Where else would it come from? Forest spirits whispering to the model?) The 'availability' is made evident by training a linear regression for classification on the learned representation (keeping the base network fixed). This works decently-well for a good representation, and not at all for raw pixels. The point is that you can use 'simple' algorithms - including kNN - on this transformed view of the inputs; it's not magical at all. FWIW, I find the classification problem space to be riddled with conceptual holes, while other application areas are much less problematic. The use of neural networks for compression makes a lot of sense, and much of representation learning can be thought of as augmentation-invariant compression for some family of augmentations. Some of the recent big advances are really about getting away from the problems of classification. Meanwhile, reinforcement learning, diffusion learning, recurrent networks, and GANs all bring different kinds of 'interactivity' into the learning problem, and resolve your complaint about ML algorithms working on 'just data.' The iterative processes in these models provide space to push against the data with decisions, and react accordingly. I find your complaints to be lacking in curiosity or even knowledge about what people are actually doing in the field... Great results keep piling up and ML systems are used by billions of people every day, while you continue to call no-true-scotsman...
- mjburgess 5y ago
  Representations in the animal sense can be observed as that which is in common across inferential contexts inn which animals are confronted by the need to use them: My model of "pen" can be observed in action when: I write with a pen; I'm asked to pass a pen; when I'm asked "what color pen do you like to write with?" and so on. What is this structure "pen"? Is it a latent pattern within my experience? No! It is a means of regulating my body, imagination, emotion, etc. "The pen is mightier than the sword!" is something I can agree, or disagree with; I have a feeling about it. And in inferential contexts which require me to feel, I feel. This is what animals acquire when they understand the world. This is what everyone believes ML is uncovering, whether they realise it or not. We just dont have these ML-like templates, so we don't realise how thin and brittle they are. When ML is run on text, is detects character patterns which form "character templates" that other text can be compared to. I cannot ask, sincerely, what the machine thinks about whether "the pen is mightier than the sword" -- because it has no attitude towards that proposition. The character structure of the phrase "the pen is mightier than the sword" relative to a body of text, has nothing to do with any concepts -- that structure has nothing to do with the inferential contexts in which those concepts will become useful. Rather, this thin character-space text structure can be repeated to people in the form of an illusion: we the dumb apes who have the concepts are fooled into thinking that some machine assembling of these words had something to say. This is a very cheap trick, and fails pretty trivially. The relevant structure to be learnt, the representation, has nothing to do with sequences of characters. Though the industry likes to talk about "latent structure" in this data, it isnt latent. It is right there on the surface, indeed: it just is that surface. When the machine is challenged to do anyything other than report this average surface structure, it fails. ML systems dont work. They improve ROI on the margins, but automating basically broken inferential rules at scale -- it is the automation here, and speed that is useful. Not the actual quality of inference.
XuMiao 5y ago
From the physics point of view, an intelligence is to keep the universe running in the perfect order, no waste of energy to increase the temperature. All accurate predictions align our current states to the future. If we look deep enough, we always find information coming from the past and the future. The only trouble is that our computation power is bounded, and computing itself generates heat waste. Human always feel that we have free will to design. It's likely a mistake in our intelligence unless the universe is infinitely dimensional where we can process the energy flow in any way we want.
pizza 5y ago
Maybe we're talking about different mathematical formalisms.. When you say latent representational structure not present in the data, am I correct in thinking you mean: argmin theta: Loss = dist(data, model(data)) where model(data) = layerL( ... layer3 ( layer2 ... layer1(data) )) is actually obscuring some conditioning factor human_understanding(observed data) which actually goes into all the layers (by their choice of design, initialization), distance functions, optimization steps, data cleaning, splits, etc.? And furthermore, also that P(model simply memorizes observations + human provided inductive cues | model finds structure) >> P(model does something beyond memorization of observations + human cues | model finds structure) ? What would be your test to determine this? For instance, how would you measure the information content (under some pragmatic-enough encoding that could be realizable and built today) of the memorization + human cues, vs memorization + human cues + the extra bit that the model concludes? Not a gotcha question, curious about your answer. Though, I have to say something about the first bit. Suppose you were comparing the safety and probability of catastrophic failure of some of the earliest buildings made in human civilization compared to something made in 2022. And suppose you conditioned that probability on the sum total of human knowledge at the time. The same laws of physics apply, but different amounts of knowledge went into producing each artifact. Can we say that the latent structure (human theory of engineering) in the modern building (model) reveals more about the ground truth of the world (laws of physical reality) than is inside of the observations data set (does it collapse or not) than the old model? I think we can. And I don't feel that embarrassed to say that deep learning is just a part of the wider landscape of human experience. You may be interested in this article that proposes Rip van Winkle's Razor: if you apply a state of the art method to an old dataset, for a fair comparison, you need to measure and take into account the increase in tooling that has become available since the time the benchmark was first released: http://www.offconvex.org/2021/04/07/ripvanwinkle/ http://www.offconvex.org/2021/04/07/ripvanwinkle/
- mjburgess 5y ago
  Yes, I think that's largely right. What we require is a "bridge" from any particular feature space to the target. A compression of a feature space doesn't produce this bridge. When we look at a photo, its ink-space arragement can be bridged to its depicted situation via our ability to brigde. We see in the ink, a world. We do this via concepts which bridge. We get those by being in the world. We can "roughly accept" limited forms of inference which simply operate within a feature space, ie., we can be fooled by photos "made on photoshop" which appear to depict a situation which never really existed. A machine operates in "inkspace" and we rig it only ever to produce "human-bridgable" arragements of ink. The machine itself doesnt draw a situation -- as with an artist painting a room. A machine isnt in the room. It is, as-if it were in the room, and as if it had painted it. This illusion is created by us, we rig the system to average historical paintings of rooms, and only ever show the apes-at-home a good picture. They think, looking at it, that the machine has painted something. One formulation of this, I suppose, is to add "| HumanUnderstanding(Y-as-labelled, Y-the-actual-target)" to everything
sjg007 5y ago
I haven't read the paper yet but you present an interesting argument to think about but I am not sure that neural network based approaches are mutually exclusive to experimental design and can in fact inform them. I'd love to talk more about this. Is anyone interested in a HN journal club?

nkozyra 5y ago

It sounds like the unique attribute you're describing in humans is nuance and contextual understanding, but we developed those as a way of doing something a neutral network does inherently by means of error adjustment. I don't think there's ever an expectation that the data is "truth" but that we have to contextualize it and adjust it. Humans adapted to those through different mechanisms than NN but the goal is still fundamentally the same: to predict something with as much accuracy as possible despite the infinite noise of the data that is the world.

mjburgess 5y ago
Alas the goal isnt the same, and here "predict" is the culprit. What ML systems do is offer statistical estimates, which is one very narrow meaning of "predict". When I "predict" what will happen when a glass falls, I'm not giving you any statistical estimates. I'm running a mental simulation of the world in its relevant parts and using this to predict what will happen. The heart of basically almost all knowledge of the world is actually counterfactual (ie., requiring simulation), it's always "if this happens (in the simulation), this will happen (in the simulation)". To acquire this ability to predict one has to understand, ie., to have the right concepts, reason with them and so on. When a person "has the concept `7`" they can reason thus: "6 eggs would be fewer", "7 is an odd number", "7 is a whole quantity", "with 7 less of 10, i'd have 3" etc. Ie., `7` is available to play a fundamental role in simulating scenarios and reasoning with them. When a NN produces a mapping from {Images} -> {Labels} such that the squiggle `7` produces the label `"7"` it doesnt do so via prediction in the meaningful sense of the world. Rather it has found a means of averaging sqiggles, and compares one squiggle to its averages and reports what the historical similar squiggles were labelled. This is a game which is only useful because we, who can reason with `7` profoundly about the world (etc.), have produced datasets which express our knowldge of the world in ways that dont require it. Ie., we present to the machine something whose average is useful. The machine is not trying to, nor does it, nor can it, predict. It has no knoweldge at all. It is just a correspondance table we have provided.
- daveed 5y ago
  I hear what you're saying, but I would not state so confidently what prediction "is" and what concepts "are", in the human model.
  - mjburgess 5y ago
    I think this conversation gets away from itself very quickly if you run with generic terms: data, prediction, and so on. People think that NNs/ML/etc. works, and now the relevant questions are philosophical: if it works, how. But it doesnt work. In very silly trivial ways. It's only taking averages over historical data. And all it can do is that. As soon as you are alive to what thinking about things requires (ie., a live embeddedness into an environment, imagining scenarios, etc.) it's trivial to expose the magic show. It isnt philosophical, it's quite literally just broken and doesnt work. I cannot ask GPT, "do you like what i'm wearing?", or "what would it take to change your mind on drug legalisation?" -- and so on. Indeed, I can't ask it, "if I have seven eggs, should I make a cake with 6 and keep the 7th -- or should I just use 7?". I can't ask it anything which actually requires it to reason with concepts, because it doesnt have any; and I cant ask it about anything around now because it is just summarising what text has already been produced. The whole AI hype industry has a vested interest in the philosophy here, ask its premised on everythign working. Almost nothing works. The idea that GPT presents any philosophical challenges is nearly absurd; about as many as a shredder at a library.

clircle 5y ago

"We have more data than ever, more good data than ever, a lower proportion of data that are good, a lack of strategic thinking about what data are needed to answer questions of interest, sub-optimal analysis of data, and an occasional tendency to do research that should not be done."

youhavetosayit 5y ago

<< almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to "be in the world" to resolve; and resolution is a dynamic process which you "have to be here for". >> Sounds like a magic view of the mind to me. It's suspicious that this is never said about the function of Liver or Kidneys, only about the function of the Brain. Maybe the brain is protecting itself, like if you ask a cow about steaks.

mjburgess 5y ago
The brain doesnt do it, the body does. The way we can tell if a stick is bent, or not, in a fluid is to remove the stick. Animals arent magic, theyre just actually present in the world. The converse is true: if the world's structure could be uncovered by data analysis, then it would be something like The Matrix; and the whole history of science would be falsified. It turns out the hardwork of experiment was never required, one could just blindly measure anything in any way, and somehow, knowledge would be produced.
- lifeisstillgood 5y ago
  So (trying to grok this) in order to "know" I have picked the box up, I can feel its weight, wave my hand in the empty space etc. I am "in the world" and use side channels to build my own reward function for my own eyesight. (presumably this is the first 9+ months of life. if I only had eyesight i could not build a reward function ? Does this mean AI could do this if they had other sensors like weight gauges in the servos?
  - mjburgess 5y ago
    Well the body has properties which enable it to formulate concepts which begin simply as motor techniques. It is unlikely that servos have the right properties, but it's an open question. I strongly suspect the body's ability to organize itself at the cellur-to-organ level over time, ie., to grow, is an essential component of how we develop & deploy novel motor techniques. It is these techniques which we use to resolve the ambiguity inherent within any system of measurement -- we, of course, learn to see -- but i think "learn" here is really a complex form of motor control which ends up structruing our perception and then reflexivily, we build on that to develop more motor techniques. We learn to see in the sense that we first learn to move, then we see better, and we cycle until we're building electron microscopes. Explicit cognition is really just a book-keeping/accounting system to tidy up this process. In modelling only this, we're relying on "doing all the hardwork" for machines. Intelligence isnt what cognition does when it tidy's up your concepts, its the very having of those concepts. Cognition is a big, largely neurotic unself-aware misdirection. It's what academics think is important -- failing to notice they have any body at all. They only got their cherished abstractions by first moving their meat -- and whilst that can be dropped for the most pure and formal of ideas (eg., arithmetic) -- it cannot be dropped for discovery as such.

jah242 5y ago

There might be some truth in what you say for very large image and language models that use supervised learning. It is really hard to see how this 'it's just a lot of good data' view applies to deep reinforcement learning where the model learns multi step policies from raw input data (e.g a camera on a robot) with only a rough high level reward function to guide it. If therefore (as seems to be the case) you can abstract the information humans need to provide to the model/learning system to ever high levels of reward function (and thereby vastly reduce the information provided by humans) then it seems very hard to argue that the model (and the training process) isn't doing to some degree what you describe as: 'incredible amounts of experimental work to carve-the-world along its joints, ie., to have the right concepts; and incredible amounts of work to measure along its joints, ie., to have the right units. And then to eliminate all the coincidences and irrelevances.' For example, imagine a robot learning from scratch to pick objects up based on raw pixel data with only a scalar reward function - where in this process is the human preparing the data so the model only has to average?

mjburgess 5y ago
> For example, imagine a robot learning from scratch to pick objects up based on raw pixel data with only a scalar reward function - where in this process is the human preparing the data so the model only has to average? Great -- so do you have an example of such a system? I'd be inclined, initially, to deny that it exists. If your reward function expresses a reward for the goal of "picking up objects with (pixel-space) properties etc.", you're cheating. In this case, the reward function serves the role of the data: ie., prepared by us to work. Indeed, a function is just a dataset -- and the reward function here is being sampled by the system. You'd need to show me a system whose reward function / dataset didnt "contain the solution", in the manner of animals who respond to the world without already having all the information about it. The relevant capacity a system needs to have, in both cases, is being able to take a profoundly ambiguous environment and produce a dataset/reward-fn which "carves along its joints". Ie., which effectively eliminates that ambiguity. When such ambiguity & coincidence is eliminated, there's basically nothing left to do -- it's that basic nothing which we task machines with doing. Ie. running `mean(sample(unambigious relevant well-carved data))`. You'll note its the *properties* of the data which express intelligence & learning.

euphetar 5y ago

As an ML Research Scientist, I have never heard this interpretation. It's a very interesting thought that NN == kNN. It puts some of my lingering intuitions in clear wording. Thank you for this. I think you are close to truth. This would explain why even the largest language models can't generalize beyound the training set. At the same time, I disagree that analysis is out of proportion. It might be some clever averaging, but it does useful and interesting things. Take a look at Google: some clever averaging can get you a long way. It would be great to understand how it works and how can we go beyond it. I do believe we need a paradigm shift, but it does not come out of nothing.

mjburgess 5y ago
I am myself trying to get closer to a clear formulation of this problem, which is why I'm writing here. Here's what I have so far, ML systems (eg., NN) remember averages (, compressions) of historical data. They are useful, wrt the problem, iif (1) the problem's target function exists; (2) the data is relevant, unambiguous, well-carved; and if (3) these properties will hold regardless of likely permutations to the problem's framing. Systems are given data with these properties by significant amounts of experimental design, work, and effort by people. Absent these properties, data is useless. Producing data with these properties requires intelligence, and no machine systems exist which can do it. My issue with research into ML on the whole, is that it *assumes* these properties and then explains how the systems work. I understand why this is interesting from a formal perspective... but it fails to note that this situation is almost never how ML is used. There is no function from Image->Animal, ie., biologists arent just idiots who could have just looked at some pixel patterns. Pixel patterns are radically ambigious wrt to `Animal`, and so even an infinite sampling of (Image, Animal) is not enough for ML. ... so what on earth are ML systems doing? This is a bigger research question: to characterise how ML performs when this assumed setup fails. And you know, that research almost doesnt exist. This is an industry led by partisans to its success. What do you think would happen if research actually talked about the dynamics of ML systems performance when (1) the target doesnt exist; (2) data isnt relevant & unambigious; (3) the problem framing will permute most times its deployed.... Suddenly we'd have an explaination of why 2016 wasnt the year self-driving cars were delivered. And indeed, likewise, of why even 2036 wont be.

vegesm 5y ago

I don't really get the first part of your comment. You say an NN is "just" compression + kNN and does no representational learning. But finding a compression (a transformation in other words) that makes kNN feasible on the data is exactly what people mean when they say it finds a hidden representation. It is a highly non-trivial task: e.g. simple distance in pixel space between images would get you useless results.

mjburgess 5y ago
People have the notion that a latent representation in the animal sense, ie., a concept -- is the same thing as your "representation" in the NN sense. That's not the case. You're right that if I find a predictive compressions of faces, say F1...n then they arent literal "rememberings". And they seem to be able to participate in a decision process (eg., classification) which doesnt seem to target pixel patterns. However I think this is kind of an illusion. What `F1..n` are, are ambiguous pixel-space projections of the abstraction which isnt present in this projection. When I have the concept "this type of face" I can reason with it beyond similarity in pixel-space. When we form representations we arent restricted to reasoning with them in only one space (eg., how faces look as pixels). We (perhaps superstitiously) impart to machine "representations" an actual depth which they lack. They are templates derived from the spaces they live in, eg., pixel-space; and have only the properties that space affords (eg., pixel-geometry). Reasoning beyond that space, and those properties, doesnt work. People think it does. This is the illusion. Templates derived from this data, that we provide, function like actual representations because we simplify the world for the machine -- and prepare its environment so that its pixel-space templates are good enough.
- PaulDavisThe1st 5y ago
  I've been with you so far, but .. > When I have the concept "this type of face" I can reason with it beyond similarity in pixel-space. I think there are at least two possible things that might going on here: 1. we're "trained" on non-pixel data (to use the same framing) and so it seems obvious that we would reason with a concept like "this type of face" in a non-pixel space. 2. the experience of "reasoning with it" is an illusion, and is merely the subjective experience that we have when our brains do stuff with whatever their underlying representation is. That is, we may have no real knowledge of what space our own "face model" is built on, what it represents, what properties it considers.

mjhirn 5y ago

I have been working and thinking about data and ML models for many years and this is probably the most concise and lucid summary of a post NN philosophy I have heard so far. It also resonates with me. Thanks for sharing.

s_Hogg 5y ago

I like what you're saying, but doesn't this just reduce to "garbage in, garbage out"?

mjburgess 5y ago
Well if you, like me, see ML as only a garbage shaft, then yes. But a lot of people think of it as a fancy processing plant which can take sewage and make diamonds. Really, it takes carefully prepared gemstones and crushes them into a shiny powder.

zmgsabst 5y ago

This is self-righteous nonsense. DNNs end up encoding parts equivalent to type theories in their operations — extracting an “internal language” (in the TT/CAT) sense which models the data. In particular, convolutions are a form of extracting/deducing types from your input. A full DNN deduces a model of the data in which it can type the incoming data and then reason about relations on those types to decide the challenge. What is it that you think you’re doing, besides forming a model from data in which you reason based on an internal language, which describes the data you’ve seen? Your post comes across as ignorant of eg, the overlap between TT and DNNs, while simultaneously full of woo about how humans operate.

mjburgess 5y ago
Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. What I think we're doing is, you know, forming concepts which are not constrained to spaces. I dont even really think our concepts, which are mostly just bundles-of-techniques, live within the space of the target object, nor its measurment space (ie., that of experience). Eg., my "pen" includes being able to grasp a pen (indeed, the relevant motor skills are prior-to and necessary-for the abstract concept formation). What you're talking about is a pixel-space projection of "pen" being able to function like the (animal, body-first) concept "pen". It doesnt. Rather "pen" in pixel space is, sure, an "internal langauge" of a sampling of AllPixelPatterns concerned with pixel-space projections of "pen". This i'd call a "template", and I dont think it has almost anything to do with concepts/representations/etc. What we are doing when we acquire motor techniques which produce "ways of sensing and moving" that eventually could become reified as the abstract "Pixel Pen" is really nothing at all like sampling PixelPenSpace and deriving a tempalte. To see the difference, consider that our concept allows us to resolve ambiguities -- eg., if I think something might be a pen, I can go beyond one space of measurement (eg., sight) -- eg., move the pen, write with it etc. -- and thus return to the target space "Is Pen?" with sufficient confidence. Once such "mere templates" exist, everything we do, isnt required. Indeed, a calculator can take over at that point; put it in a cupboard forever to repeat whatever thin inferences pixel templates admit.
- zmgsabst 5y ago

axg11 5y ago

NN == kNN is a really interesting thought, but I think it’s the wrong framing. All learning is bound by kNN, including, I would argue, human intelligence. When we compare and contrast things, be it words, objects, or images, the very best we can do is to compare those “units” to all other examples that we have ever observed. There is no such thing as generalising beyond kNN. The real magic in generalising is not how many nearest neighbours you are able to compare against, the magic is in _which_ qualities you use to determine the kNN. Another way to state this is: intelligence is in the distance metric/embedding/compression, not the ability to process all examples. When we say that humans are able to generalise, what we really mean is that we are able to compare and contrast specific subsets of qualities for a given object against all other things objects we have knowledge for. The way we choose and weight those specific qualities is what results in our “intelligence”. So when we say that NN == kNN, I think this is actually a really impressive achievement for neural nets. Neural networks are able to produce kNN level results with orders of magnitude lower compute. Humans too, are able to do this, and if we’re being very thorough, we sometimes fall back to kNN-like processes

mjburgess 5y ago
Intelligence is in the properties of the data supplied to the kNN. Not in its distance function. Yes, kNN is the bound if you formulate learning as: there exists an f_true: Pixels->Animals; we have an infinite sampling of (Pixels, Animals); kNN performs at 100%. Therefore kNN is the bound. Whereas I dont think this has much to do with learning. Learning for me is: the function probably doesnt exist, we dont know what its domain would even be, and we have no idea how to sample that space. You know: the history of human civilization, but also: the history of a day of any animal's life. We encounter problems daily whose functions can only be modelled by partial input and output domains, and our job is to find ways of resolving the inherent ambiguities in those situations. Ie., to find what parts of the world are relevant, to find ways of measuring it, and hence to find more data until the ambiguity resolves itself. kNN being a perfect algorithm is a catastrophe for this whole formulation of AI. All it means is that the whole field of machine learning has as its limit what you can do if you already know the answer. You can see from this that it isnt learning at all. The situation is exactly backwards: ML assumes relevant well-carved unambiguous data. The whole problem of learning is precisely to produce it!

canjobear 5y ago

If neural networks do compression, then they are implicitly providing structure not present in the data. Given a dataset there are always many different ways to compress it, each corresponding to some implicit distribution over unseen datapoints. Most of these distributions don’t correspond to real-world generalizations. Yet neural networks do often—not always—choose models for compression that correspond to real generalizations. This is nontrivial.

vladf 5y ago

This comment is making a "slight of hand" itself: how do we define the metric for kNN? That will directly impact kNN performance. Using Euclidean norm over pixels on imagenet will not get you anywhere. By recasting classification into a kNN lens you're basically (in the large data regime) recasting the problem to kernel learning under some underspecified RBF kernel with an unknown "proximity" metric instead of Euclidean. In grad school, someone in my lab actually tried scaling kernel methods directly to imagenet too. I don't think that ended up working naively (an interesting neural variant worked for CIFAR10, though [1]). If all NNs are doing is efficiently learning an appropriate kNN metric (or, more idiomatically and generally put, learning kernel parameters for some implicit neural kernel), that's still really powerful and all we've done is just renamed "learning representations." [1]: https://arxiv.org/abs/2003.02237 https://arxiv.org/abs/2003.02237

dotnet00 5y ago

I think NN=kNN is reductive to the point of near uselessness as a statement. You can treat human object recognition as a sort of kNN as well in that we 'detect' a large number of features, the grouping of which ultimately allows us to recognize something. But of course that doesn't actually tell us much about how we recognize things, since that explanation just changes the question to "how do we detect useful features?". Similarly, I think most ML people recognize that NN classifiers are just learning to look for groups of features, yet that doesn't really count as an analysis because what we want to understand is why it arrives at the features it does. >The magic of ML is a sleight-of-hand trick -- we dont really need to know how its averaging of our data does anything useful -- it almost never does. I strongly disagree, we do need to know how the 'averaging' does what it does so we can improve our models further. A lot of big steps in NN architectures have come in part from applying theories and insights regarding the properties of the training process, so a deeper understanding of the process may allow us to make better models, ones that have better accuracy as well as ones that train faster with less data. >Rather, it is our "experimental design" which produces the usefulness of the system. ML algorithms are just interpolations and averages through data prepared to produce useful averages by (literally millenia) of human ingenuity. >It takes actual intelligence to do this because the world isnt data, and almost any measurement one cares to make (with eyes, even) produces endless ambiguities and coincidences that you have to "be in the world" to resolve; and resolution is a dynamic process which you "have to be here for". I think most of the industry agrees that current ML models are sort of like being able to make the icing but not the cake (i.e AGI), so I'm not really sure what you're trying to say. Of course the models need a general intelligence to filter general data down to a specific problem and set things up for the model to learn?

djoldman 5y ago

My hot take is that this correct in spirit. However, it just moves the conceptual line between data/feature engineering and modeling so as to make the data/feature engineering encompass nearly all of the process. By this definition of ML, one might just say that clever ML is a really clever way to engineer useful features, after which we "merely" average them. It feels like semantics.

mjburgess 5y ago
From a certain POV the structure of an animal's body, as for example, it learns to ride a bike can be given this kind of formulation. Namely, that the muscle tissue and the motor cortext express a latent representation of riding which is formed under the animal's self-regulative goals of "stability" and "getting to some location" etc. From this POV, one can model any aspect of reality with this sort of language. I used to be annoyed by this, but less so now. Annoyed because it's pretty hopelessly useless describe any complex physical system this way. It only makes sense if your relevant "feature spaces" are discrete and computationally tractable... i dont think our muscles are good candidates. (Nor likewise, is almost anything). Nevertheless, you are right that "ML" on a domain whose dimensions are infinite, and in which slices through those dimensions are infinite, and whose models are chaotic -- then ML with an infinity of time to run would produce an animal. And indeed, an intelligent system. One has to remember here that no one is doing this. What theyre doing is taking a highly regulated non-chatoic, tractable, measured and largely unambigious slice through these spaces --- sampling them --- and taking an average. Who gave you the knife to take this slice? Was it your cognition, was it the world? No, neither, it was your body -- largely; and its ability to regulate your imagination; and to guide action. Formulating reality as-if its patterns can be determined statistically leads to absurd levels of non-computability. Every possible partitioning of the world, in every possible arrangment, moving in every possible way -- what discrete model of this is there? There isn't one. And this isnt semantic. The challenge of building an intelligent system is a (bio-)engineering one: we require a material which grows intelligently, whose internal chaos is useful. Not a dumb little calculator which comes in to average our efforts after the fact.
- neatze 5y ago

sdenton4 5y ago

"Namely, that it uncovers latent representational structure not present in the data." Who thinks this? You're talking to straw-men. The latent structure is absolutely present in the data, but a series of transforms makes the structure more readily available. (Where else would it come from? Forest spirits whispering to the model?) The 'availability' is made evident by training a linear regression for classification on the learned representation (keeping the base network fixed). This works decently-well for a good representation, and not at all for raw pixels. The point is that you can use 'simple' algorithms - including kNN - on this transformed view of the inputs; it's not magical at all. FWIW, I find the classification problem space to be riddled with conceptual holes, while other application areas are much less problematic. The use of neural networks for compression makes a lot of sense, and much of representation learning can be thought of as augmentation-invariant compression for some family of augmentations. Some of the recent big advances are really about getting away from the problems of classification. Meanwhile, reinforcement learning, diffusion learning, recurrent networks, and GANs all bring different kinds of 'interactivity' into the learning problem, and resolve your complaint about ML algorithms working on 'just data.' The iterative processes in these models provide space to push against the data with decisions, and react accordingly. I find your complaints to be lacking in curiosity or even knowledge about what people are actually doing in the field... Great results keep piling up and ML systems are used by billions of people every day, while you continue to call no-true-scotsman...

mjburgess 5y ago
Representations in the animal sense can be observed as that which is in common across inferential contexts inn which animals are confronted by the need to use them: My model of "pen" can be observed in action when: I write with a pen; I'm asked to pass a pen; when I'm asked "what color pen do you like to write with?" and so on. What is this structure "pen"? Is it a latent pattern within my experience? No! It is a means of regulating my body, imagination, emotion, etc. "The pen is mightier than the sword!" is something I can agree, or disagree with; I have a feeling about it. And in inferential contexts which require me to feel, I feel. This is what animals acquire when they understand the world. This is what everyone believes ML is uncovering, whether they realise it or not. We just dont have these ML-like templates, so we don't realise how thin and brittle they are. When ML is run on text, is detects character patterns which form "character templates" that other text can be compared to. I cannot ask, sincerely, what the machine thinks about whether "the pen is mightier than the sword" -- because it has no attitude towards that proposition. The character structure of the phrase "the pen is mightier than the sword" relative to a body of text, has nothing to do with any concepts -- that structure has nothing to do with the inferential contexts in which those concepts will become useful. Rather, this thin character-space text structure can be repeated to people in the form of an illusion: we the dumb apes who have the concepts are fooled into thinking that some machine assembling of these words had something to say. This is a very cheap trick, and fails pretty trivially. The relevant structure to be learnt, the representation, has nothing to do with sequences of characters. Though the industry likes to talk about "latent structure" in this data, it isnt latent. It is right there on the surface, indeed: it just is that surface. When the machine is challenged to do anyything other than report this average surface structure, it fails. ML systems dont work. They improve ROI on the margins, but automating basically broken inferential rules at scale -- it is the automation here, and speed that is useful. Not the actual quality of inference.

XuMiao 5y ago

From the physics point of view, an intelligence is to keep the universe running in the perfect order, no waste of energy to increase the temperature. All accurate predictions align our current states to the future. If we look deep enough, we always find information coming from the past and the future. The only trouble is that our computation power is bounded, and computing itself generates heat waste. Human always feel that we have free will to design. It's likely a mistake in our intelligence unless the universe is infinitely dimensional where we can process the energy flow in any way we want.

pizza 5y ago

Maybe we're talking about different mathematical formalisms.. When you say latent representational structure not present in the data, am I correct in thinking you mean: argmin theta: Loss = dist(data, model(data)) where model(data) = layerL( ... layer3 ( layer2 ... layer1(data) )) is actually obscuring some conditioning factor human_understanding(observed data) which actually goes into all the layers (by their choice of design, initialization), distance functions, optimization steps, data cleaning, splits, etc.? And furthermore, also that P(model simply memorizes observations + human provided inductive cues | model finds structure) >> P(model does something beyond memorization of observations + human cues | model finds structure) ? What would be your test to determine this? For instance, how would you measure the information content (under some pragmatic-enough encoding that could be realizable and built today) of the memorization + human cues, vs memorization + human cues + the extra bit that the model concludes? Not a gotcha question, curious about your answer. Though, I have to say something about the first bit. Suppose you were comparing the safety and probability of catastrophic failure of some of the earliest buildings made in human civilization compared to something made in 2022. And suppose you conditioned that probability on the sum total of human knowledge at the time. The same laws of physics apply, but different amounts of knowledge went into producing each artifact. Can we say that the latent structure (human theory of engineering) in the modern building (model) reveals more about the ground truth of the world (laws of physical reality) than is inside of the observations data set (does it collapse or not) than the old model? I think we can. And I don't feel that embarrassed to say that deep learning is just a part of the wider landscape of human experience. You may be interested in this article that proposes Rip van Winkle's Razor: if you apply a state of the art method to an old dataset, for a fair comparison, you need to measure and take into account the increase in tooling that has become available since the time the benchmark was first released: http://www.offconvex.org/2021/04/07/ripvanwinkle/ http://www.offconvex.org/2021/04/07/ripvanwinkle/

mjburgess 5y ago
Yes, I think that's largely right. What we require is a "bridge" from any particular feature space to the target. A compression of a feature space doesn't produce this bridge. When we look at a photo, its ink-space arragement can be bridged to its depicted situation via our ability to brigde. We see in the ink, a world. We do this via concepts which bridge. We get those by being in the world. We can "roughly accept" limited forms of inference which simply operate within a feature space, ie., we can be fooled by photos "made on photoshop" which appear to depict a situation which never really existed. A machine operates in "inkspace" and we rig it only ever to produce "human-bridgable" arragements of ink. The machine itself doesnt draw a situation -- as with an artist painting a room. A machine isnt in the room. It is, as-if it were in the room, and as if it had painted it. This illusion is created by us, we rig the system to average historical paintings of rooms, and only ever show the apes-at-home a good picture. They think, looking at it, that the machine has painted something. One formulation of this, I suppose, is to add "| HumanUnderstanding(Y-as-labelled, Y-the-actual-target)" to everything

sjg007 5y ago

I haven't read the paper yet but you present an interesting argument to think about but I am not sure that neural network based approaches are mutually exclusive to experimental design and can in fact inform them. I'd love to talk more about this. Is anyone interested in a HN journal club?