3 ms·

This is self-righteous nonsense. DNNs end up encoding parts equivalent to type theories in their operations — extracting an “internal language” (in the TT/CAT)

This is self-righteous nonsense. DNNs end up encoding parts equivalent to type theories in their operations — extracting an “internal language” (in the TT/CAT) sense which models the data. In particular, convolutions are a form of extracting/deducing types from your input. A full DNN deduces a model of the data in which it can type the incoming data and then reason about relations on those types to decide the challenge. What is it that you think you’re doing, besides forming a model from data in which you reason based on an internal language, which describes the data you’ve seen? Your post comes across as ignorant of eg, the overlap between TT and DNNs, while simultaneously full of woo about how humans operate.

mjburgess 5y ago
Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. What I think we're doing is, you know, forming concepts which are not constrained to spaces. I dont even really think our concepts, which are mostly just bundles-of-techniques, live within the space of the target object, nor its measurment space (ie., that of experience). Eg., my "pen" includes being able to grasp a pen (indeed, the relevant motor skills are prior-to and necessary-for the abstract concept formation). What you're talking about is a pixel-space projection of "pen" being able to function like the (animal, body-first) concept "pen". It doesnt. Rather "pen" in pixel space is, sure, an "internal langauge" of a sampling of AllPixelPatterns concerned with pixel-space projections of "pen". This i'd call a "template", and I dont think it has almost anything to do with concepts/representations/etc. What we are doing when we acquire motor techniques which produce "ways of sensing and moving" that eventually could become reified as the abstract "Pixel Pen" is really nothing at all like sampling PixelPenSpace and deriving a tempalte. To see the difference, consider that our concept allows us to resolve ambiguities -- eg., if I think something might be a pen, I can go beyond one space of measurement (eg., sight) -- eg., move the pen, write with it etc. -- and thus return to the target space "Is Pen?" with sufficient confidence. Once such "mere templates" exist, everything we do, isnt required. Indeed, a calculator can take over at that point; put it in a cupboard forever to repeat whatever thin inferences pixel templates admit.
- zmgsabst 5y ago
  > Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. This is incorrect: You can swap encoders and final fully connected sections, then mildly retrain the network to substantially save on training effort — “transfer learning”. Further, you can compose networks, eg image recognition on top of something extracting a structural map. This implies that the structures they’re “learning” generalize to different contexts — with a bit of retraining. In much the way your knowledge of programming transfers between languages (swapping encoders) and tasks (swapping final connected network). Your point seems deeply based on this incorrect understanding of DNNs.
  - mjburgess 5y ago
    Sure, as set up by the way we have constructed the dataset. "Green" isn't near "Leaves" because the world placed them there. All these properties follow from properties of the data which we have structured to produce these solutions. We arrange the target domain of one dataset up, so that the target domain of another is aligned --- and in doing so, solve the only problem which requires intelligence. Yes, the character-space structure of Y labelled stuff aligns with the pixel space structure of Y labelled stuff. And feature space templates can likewise be associated. This is just more of the same. King +Woman isn't Queen --- it isn't anything. This isn't reasoning. And likewise PicOfKing+PicOfW isnt PicOfQueen. This is schizophrenic pseudoscience --- it's actually the mind of a schizophrenic who obtains sequential thoughts just by non-semantic associations. The latent structure being learnt in both cases is just the coincidences we rig in the Y domain --- Inhave no doubt they 'transfer', we wrote them in
    - zmgsabst 5y ago
      We don’t write them in though. You can show a DNN a large sample of photos of animals, retain the lower sections of the network, and then show it a large sample of landscapes — and it will be able to recognize those faster precisely because a well-trained network picked up on the latent structure of shapes, and can reason about shapes in its internal language for describing pictures. It’s acquiring genuine semantic information — and learning about new experiences relative to that previous acquired understanding. We didn’t create that similarity of form in nature — and DNNs discover that in much the way we do: finding recurring, abstract patterns to utilize in our internal language.
YorkshireSeason 5y ago
Could you elaborate on the connection between Deep Learning and type theory, resp. topos theory that you allude to? It sounds fascinating. At the some time, I'm slighly surprised that I am not aware of results in this direction.
- zmgsabst 5y ago
  I think it mostly exists at the level of folk lore right now, with preliminary bridges. I’ve spoken some with people at, eg HoTT 2019 or Topos Institute speakers, but I think the state of publishing is that the other direction is being worked out fully — since it solves problems in the math community like accelerating proof engines via GPUs. (Maybe, I hope.) There’s three approaches to seeing the connection: 1. If the manifold hypothesis is true, then the semantic information between the original manifold and reduced manifold will be in the topology of that map — and your TT is a relabeling of the synthetic topology description. 2. For the convolution comment, take a TT statement, view it as a CuTT structure, and draw the construction diagram of that “shape”. Project that into a matrix encoding and… convolutions are right-side type division! Eg, finding a submatrix that matches “Line(…)” in your overall matrix. But it’s right-sided division since you can only remove the “innermost” factor in a product that way. (There’s also a left-handed, but it’s less relevant here.) 3. The paper on ML via hyper plane splits to create data coverings also suggests that what we really want to know is the topological relationships between kinds of data. This is furthered by, eg, “animal” in word2vec ideally being the centroid of the open ball covering animals, as they’re embedded.
  - mjburgess 5y ago
    The manifold hypothesis is true when the measurement bases aren't orthogonal, and there's lots of them eg., when we measure 3D objects by their relative position to 100 objects in a room. That isnt a bridge principle between spaces which measure different properties. The low-D structure of any high-D dataset is only a "washing out" of spare measurements of the same properties. These arent different "property-spaces", ie., the low-D structure isnt a world one has bridged to via statistics (or grad desc, etc.). The properties the formal structure of text captures aren't the properties of objects.. they're something like the social-vocab-coincidental usage of language users in how they "orient their words" around topics. If we change our usage, models which find this structure, become invalidated. We language-users routinely change this structure, and robustly against misunderstanding, because we bridge to the relevant property spaces by being in the world.

mjburgess 5y ago

Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. What I think we're doing is, you know, forming concepts which are not constrained to spaces. I dont even really think our concepts, which are mostly just bundles-of-techniques, live within the space of the target object, nor its measurment space (ie., that of experience). Eg., my "pen" includes being able to grasp a pen (indeed, the relevant motor skills are prior-to and necessary-for the abstract concept formation). What you're talking about is a pixel-space projection of "pen" being able to function like the (animal, body-first) concept "pen". It doesnt. Rather "pen" in pixel space is, sure, an "internal langauge" of a sampling of AllPixelPatterns concerned with pixel-space projections of "pen". This i'd call a "template", and I dont think it has almost anything to do with concepts/representations/etc. What we are doing when we acquire motor techniques which produce "ways of sensing and moving" that eventually could become reified as the abstract "Pixel Pen" is really nothing at all like sampling PixelPenSpace and deriving a tempalte. To see the difference, consider that our concept allows us to resolve ambiguities -- eg., if I think something might be a pen, I can go beyond one space of measurement (eg., sight) -- eg., move the pen, write with it etc. -- and thus return to the target space "Is Pen?" with sufficient confidence. Once such "mere templates" exist, everything we do, isnt required. Indeed, a calculator can take over at that point; put it in a cupboard forever to repeat whatever thin inferences pixel templates admit.

zmgsabst 5y ago
> Sure, but all this is only within one space. The "internal language" is a formalism of that space, and not beyond. This is incorrect: You can swap encoders and final fully connected sections, then mildly retrain the network to substantially save on training effort — “transfer learning”. Further, you can compose networks, eg image recognition on top of something extracting a structural map. This implies that the structures they’re “learning” generalize to different contexts — with a bit of retraining. In much the way your knowledge of programming transfers between languages (swapping encoders) and tasks (swapping final connected network). Your point seems deeply based on this incorrect understanding of DNNs.
- mjburgess 5y ago
  Sure, as set up by the way we have constructed the dataset. "Green" isn't near "Leaves" because the world placed them there. All these properties follow from properties of the data which we have structured to produce these solutions. We arrange the target domain of one dataset up, so that the target domain of another is aligned --- and in doing so, solve the only problem which requires intelligence. Yes, the character-space structure of Y labelled stuff aligns with the pixel space structure of Y labelled stuff. And feature space templates can likewise be associated. This is just more of the same. King +Woman isn't Queen --- it isn't anything. This isn't reasoning. And likewise PicOfKing+PicOfW isnt PicOfQueen. This is schizophrenic pseudoscience --- it's actually the mind of a schizophrenic who obtains sequential thoughts just by non-semantic associations. The latent structure being learnt in both cases is just the coincidences we rig in the Y domain --- Inhave no doubt they 'transfer', we wrote them in
  - zmgsabst 5y ago
    We don’t write them in though. You can show a DNN a large sample of photos of animals, retain the lower sections of the network, and then show it a large sample of landscapes — and it will be able to recognize those faster precisely because a well-trained network picked up on the latent structure of shapes, and can reason about shapes in its internal language for describing pictures. It’s acquiring genuine semantic information — and learning about new experiences relative to that previous acquired understanding. We didn’t create that similarity of form in nature — and DNNs discover that in much the way we do: finding recurring, abstract patterns to utilize in our internal language.

YorkshireSeason 5y ago

Could you elaborate on the connection between Deep Learning and type theory, resp. topos theory that you allude to? It sounds fascinating. At the some time, I'm slighly surprised that I am not aware of results in this direction.

zmgsabst 5y ago
I think it mostly exists at the level of folk lore right now, with preliminary bridges. I’ve spoken some with people at, eg HoTT 2019 or Topos Institute speakers, but I think the state of publishing is that the other direction is being worked out fully — since it solves problems in the math community like accelerating proof engines via GPUs. (Maybe, I hope.) There’s three approaches to seeing the connection: 1. If the manifold hypothesis is true, then the semantic information between the original manifold and reduced manifold will be in the topology of that map — and your TT is a relabeling of the synthetic topology description. 2. For the convolution comment, take a TT statement, view it as a CuTT structure, and draw the construction diagram of that “shape”. Project that into a matrix encoding and… convolutions are right-side type division! Eg, finding a submatrix that matches “Line(…)” in your overall matrix. But it’s right-sided division since you can only remove the “innermost” factor in a product that way. (There’s also a left-handed, but it’s less relevant here.) 3. The paper on ML via hyper plane splits to create data coverings also suggests that what we really want to know is the topological relationships between kinds of data. This is furthered by, eg, “animal” in word2vec ideally being the centroid of the open ball covering animals, as they’re embedded.
- mjburgess 5y ago
  The manifold hypothesis is true when the measurement bases aren't orthogonal, and there's lots of them eg., when we measure 3D objects by their relative position to 100 objects in a room. That isnt a bridge principle between spaces which measure different properties. The low-D structure of any high-D dataset is only a "washing out" of spare measurements of the same properties. These arent different "property-spaces", ie., the low-D structure isnt a world one has bridged to via statistics (or grad desc, etc.). The properties the formal structure of text captures aren't the properties of objects.. they're something like the social-vocab-coincidental usage of language users in how they "orient their words" around topics. If we change our usage, models which find this structure, become invalidated. We language-users routinely change this structure, and robustly against misunderstanding, because we bridge to the relevant property spaces by being in the world.