11 ms·

Deep Learning Is Applied Topology

508 points by theahura 1y ago

esafak 1y ago
If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry, with a metric. Just as in real life, we want to be able to compare things. Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement. Here are some relevant papers: * Topology and geometry of data manifold in deep learning (https://arxiv.org/abs/2204.08624 https://arxiv.org/abs/2204.08624) * Topology of Deep Neural Networks (https://jmlr.org/papers/v21/20-345.html https://jmlr.org/papers/v21/20-345.html) * Persistent Topological Features in Large Language Models (https://arxiv.org/abs/2410.11042 https://arxiv.org/abs/2410.11042) * Deep learning as Ricci flow (https://www.nature.com/articles/s41598-024-74045-9 https://www.nature.com/articles/s41598-024-74045-9)
- profchemai 1y ago
  Agree, if anything it's Applied Linear Algebra...but that sounds less exotic.
  - lostmsu 1y ago
    Well, we know it is non-linear. More like differential equations.
- theahura 1y ago
  > Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? If you've ever played with GANs or VAEs, you can actually answer this question! And the answer is more or less 'yes'. You can look at GANs at various checkpoints during training and see how different points in the high dimensional space move around (using tools like UMAP / TSNE). > I imagine it violently changing at first before stabilizing, followed by geometric refinement Also correct, though the violent changing at the beginning is also influenced the learning rate and the choice of optimizer.
  - esafak 1y ago
    And crucially, the initialization algorithm.
umutisik 1y ago
Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).
- motoboi 1y ago
  Your comment sits in the nice gradient between not seeing at all the obvious relationships between deep learning and topology and thinking that deep learning is applied topology. See? Everything lives in the manifold. Now for a great visualization about the Manifold Hypothesis I cannot recommend more this video: https://www.youtube.com/watch?v=pdNYw6qwuNc https://www.youtube.com/watch?v=pdNYw6qwuNc That helps to visualize how the activation functions, bias and weights (linear transformations) serve to stretch the high dimensional space so that data go into extremes and become easy to put in a high dimension, low dimensional object (the manifold) where is trivial to classify or separate. Gaining an intuition about this process will make some deep learning practices so much easy to understand.
- thuuuomas 1y ago
  I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?
  - hiddencost 1y ago
    Maybe a little less with the ad hominems? The OP is providing an accurate description of an extremely immature field.
    - cnity 1y ago
      Many mathematicians are (rightly, IMO) allergic to assertions that certain branches are not useful (explicit in OP) and especially so if they are dismissive of attempts to understand complicated real world phenomema (implicit in OP, if you ask me).
mirekrusin 1y ago
Just because manifold looks a bit like burrito if you squint doesn't mean it is a burrito.
- ComplexSystems 1y ago
  What if you don't have to squint very much?
  - mirekrusin 1y ago
    Same amount of squinting needed as for claim that deep learning is just a bunch of matrices. Or 0s and 1s. Cool. Now what?
sota_pop 1y ago
I’ve always enjoyed this framing of the subject, the idea of mapping anything as hyperplanes existing in a solution space is one of the ideas that really blew my hair back during my academic studies. I would nitpick at your “dots in a circle example - with the stoner reference joke” I could be mistaken, but common practice isn’t to “move to a higher dimension”, but use a kernel (i.e. parameterize the points into the polar |r,theta> basis). All things considered, nice article.
- theahura 1y ago
  I'm pulling directly from Chris Olah's blog post with that example. But I will say that in practice, its always surprising how increasing the dimensionality of a neural network magically solves all sorts of problems. You could use a kernel if you don't have more computation available, but given more computation adding a dimension is strictly more flexible (and is capable of separating a much wider range of datasets)
  - benl 1y ago
    Your explanation of finding a surface to separate good reasoning traces from bad reasoning traces in a high dimensional space worked as a great framing of the problem. It seems though that the surface will be fractal - the distance between a good trace and a bad trace could be arbitrarily small. If so then the work required to find and compute better and better surfaces will grow arbitrarily large. I wonder if there is a rigorous way to determine if the surface is fractal or not.
polotics 1y ago
Ok, how do transformers fit into this understanding of deep learning?
- motoboi 1y ago
  Transformers don't feel differentiable (because of the attention mechanism), but they actually are (as being back-propagation based forces it to be). The attention mechanism is not a stretching of the manifold, but is trained to be able to measure distances in the manifold surface, which is stretched and deformed (or transformed?) in the feed-forward layers.
- theahura 1y ago
  Transformers learn embedding representations of tokens, which are easily mapped into a space. Similar tokens are mapped to similar places on the space. The fully connected layer at the end of each transformer block defines a transformation of a set of points in a space to another point in that space, not unlike the example of adding colors together to get a new color
- jebarker 1y ago
  Transformers (with self-attention being the key operation) are kernel smoothers which fits easily into this view of the world. See here: http://bactra.org/notebooks/nn-attention-and-transformers.html http://bactra.org/notebooks/nn-attention-and-transformers.ht...
parpfish 1y ago
This is also how I've often thought about deep learning -- focusing on the geometry of the data at each layer rather than the weights and biases is far more revealing. I've always been hopeful that some algebraic topology master would dig into this question and it'd provide some better design principles for neural nets. which activation functions? how much to fan in/out? how many layers?
amelius 1y ago
[flagged]
- adverbly 1y ago
  Life is just applied category theory
- deleted 1y ago
  [deleted]
profchemai 1y ago
Once I read "This has been enough to get us to AGI.", credibility took a nose dive. In general it's a nice idea, but the blogpost is very fluffy, especially once it connects it to reasoning, there is serious technical work in this area (i.g. https://arxiv.org/abs/1402.1869 https://arxiv.org/abs/1402.1869) that has expanded this idea and made it more concrete.
Koshkin 1y ago
[flagged]
- deleted 1y ago
  [deleted]
fedeb95 1y ago
Interesting read. This seems hard to prove: "Everything lives on a manifold"
ComplexSystems 1y ago
I really liked this article, though I don't know why the author is calling the idea of finding a separating surface between two classes of points "topology." For instance, they write "If you are trying to learn a translation task — say, English to Spanish, or Images to Text — your model will learn a topology where bread is close to pan, or where that picture of a cat is close to the word cat." This is everything that topology is not about: a notion of points being "close" or "far." If we have some topological space in which two points are "close," we can stretch the space so as to get the same topological space, but with the two points now "far". That's the whole point of the joke that the coffee cup and the donut are the same thing. Instead, the entire thing seems to be a real-world application of something like algebraic geometry. We want to look for something like an algebraic variety the points are near. It's all about geometry and all about metrics between points. That's what it seems like to me, anyway.
- srean 1y ago
  > This is everything that topology is not about 100 percent true. I can only hope that in an article that is about two things, i) topology and ii) deep learning, the evident confusions are contained within one of them -- topology, only.
- theahura 1y ago
  fair, I was using 'topology' more colloquially in that sentence. Should have said 'surface'.
  - srean 1y ago
    Ah! That clears it up. You then mean Deep Learning has a lot in common with differential geometry and manifolds in general. That I will definitely agree with. DG and manifolds have far richer and informative structure than topology.
- steppi 1y ago
  If I had to give a loose definition of topology, I would say that it is actually about studying spaces which have some notion of what is close and far, even if no metric exists. The core idea of neighborhoods in point set topology captures the idea of points being nearby another point, and allows defining things like continuity and sequence convergence which require a notion of closeness. From Wikipedia [0] for example The terms 'nearby', 'arbitrarily small', and 'far apart' can all be made precise by using the concept of open sets. If we change the definition of 'open set', we change what continuous functions, compact sets, and connected sets are. Each choice of definition for 'open set' is called a topology. A set with a topology is called a topological space. Metric spaces are an important class of topological spaces where a real, non-negative distance, also called a metric, can be defined on pairs of points in the set. Having a metric simplifies many proofs, and many of the most common topological spaces are metric spaces. That's not to say that topology is necessarily the best lens for understanding neural networks, and the article's author has shown up in the comments to state he's moved on in his thinking. I'm just trying to clear up a misconception. [0] https://en.wikipedia.org/wiki/General_topology https://en.wikipedia.org/wiki/General_topology
Koshkin 1y ago
Speaking of: https://www2.math.upenn.edu/~ghrist/notes.html https://www2.math.upenn.edu/~ghrist/notes.html
deleted 1y ago
[deleted]
soulofmischief 1y ago
Thanks for sharing. I also tend to view learning in terms of manifolds. It's a powerful representation. > I'm personally pretty convinced that, in a high enough dimensional space, this is indistinguishable from reasoning I actually have journaled extensively about this and even written some on Hacker News about it with respect to what I've been calling probabilistic reasoning manifolds: > This manifold is constructed via learning a decontextualized pattern space on a given set of inputs. Given the inherent probabilistic nature of sampling, true reasoning is expressed in terms of probabilities, not axioms. It may be possible to discover axioms by locating fixed points or attractors on the manifold, but ultimately you're looking at a probabilistic manifold constructed from your input set. > But I don't think you can untie this "reasoning" from your input data. It's possible you will find "meta-reasoning", or similar structures found in any sufficiently advanced reasoning manifold, but these highly decontextualized structures might be entirely useless without proper recontextualization, necessitating that a reasoning manifold is trained on input whose patterns follow learnable underlying rules, if the manifold is to be useful for processing input of that kind. > Decontextualization is learning, decomposing aspects of an input into context-agnostic relationships. But recontextualization is the other half of that, knowing how to take highly abstract, sometimes inexpressible, context-agnostic relationships and transform them into useful analysis in novel domains Full comment: https://news.ycombinator.com/item?id=42871894 https://news.ycombinator.com/item?id=42871894
- mjburgess 1y ago
  Are you talking about reasoning in general, reasoning qua that mental process which operates on (representations of) propositions? In which case, I cannot understand " true reasoning is expressed in terms of probabilities, not axioms " One of the features of reasoning is that it does not operate in this way. It's highly implausible animals would have been endowed with no ability to operate non-probabilistically on propositions represented by them, since this is essential for correct reasoning -- and a relatively trivial capability to provide. Eg., "if the spider is in boxA, then it is not everywhere else" and so on
  - jvanderbot 1y ago
maxiepoo 1y ago
Isn't it more differential geometry?
srean 1y ago
The title, as it stands, is trite and wrong. More about that a little later. The article on the other hand is a pleasant read. Topology is whatever little structure that remains in geometry after you throwaway distances, angles, orientations and all sorts of non tearing stretchings. It's that bare minimum that still remains valid after such violent deformations. While notion of topology is definitely useful in machine learning, -- scale, distance, angles etc., all usually provide lots of essential information about the data. If you want to distinguish between a tabby cat and a tiger it would be an act of stupidity to ignore scale. Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations. That happens, but to claim deep learning is applied topology is absurd, almost stupid.
- theahura 1y ago
  > Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations But...you can't. The input data lives on a manifold that you cannot 'trust'. It doesn't mean anything apriori that an image of a coca-cola can and an image of a stopsign live close to each other in pixel space. The neural network applies all of those violent transformations you are talking about
  - srean 1y ago
    > But...you can't. Only in a desperate sales pitch or a desparate research grants. There are of course some situations were certain measurements are untrustworthy, but to claim that is the common case is very snake oily. When certain measurements become untrustworthy, that it does so only because of some unknown smooth transformation, is not very frequent (this is what purely topological methods will deal with). Random noise will also do that for you. Not disputing the fact that sometimes metrics cannot be trusted entirely, but to go to a topological approach seems extreme. One should use as much of the relevant non-topological information as possible. As the hackneyed example goes a topological methods would not be able to distinguish between a cup and a donut. For that you would need to trust non-topological features such as distances and angles. Deep learning methods can indeed differentiate between cop-nip and coffee mugs. BTW I am completely on-board with the idea that data often looks as if it has been sampled from an unknown, potentially smooth, possibly non-Euclidean manifold and then corrupted by noise. In such cases recovering that manifold from noisy data is a very worthy cause. In fact that is what most of your blogpost is about. But that's differential geometry and manifolds, they have structure far richer than a topology. For example they may have tangent planes, a Reimann metric or a symplectic form etc. A topological method would throw all of that away and focus on topology.
deleted 1y ago
[deleted]
terabytest 1y ago
I'm confused by the author's diagram claiming that AGI/ASI are points on the same manifold as next token prediction, chat models, and CoT models. While the latter three are provably part of the same manifold, what justifies placing AGI/ASI there too? What if the models capable of CoT aren't and will never be, regardless of topological manipulation, capable of processes that could be considered AGI? For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing. As a layman, this matches my intuition that LLMs are not at all in the same family of systems as the ones capable of generating intelligence or consciousness.
- theahura 1y ago
  Possible. AGI/ASI are poorly defined. I tend to think we're already at AGI, obviously many disagree. > For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing. I've done a fair bit of connectomics research and I think that this framing elides the ways in which neural networks and biological networks are actually quite similar. For example, in mice olfactory systems there is something akin to a 'feature vector' that appears based on which neurons light up. Specific sets of neurons lighting up means 'chocolate' or 'lemon' or whatever. More generally, it seems like neuronal representations are somewhat similar to embedding representations, and you could imagine constructing an embedding space based on what neurons light up where. Everything on top of the embeddings is 'just' processing.
  - fusionadvocate 1y ago
    I believe we already have the technology required for AGI. It perhaps is analogous to a lunar manned station or a 2 mile tall skyscrapper. We have the technology required to build it, but we don't for various reasons.
_alternator_ 1y ago
The question is not so much whether this is true—we can certainly represent any data as points on a manifold. Rather, it’s the extent to which this point of view is useful. In my experience, it’s not the most powerful perspective. In short, direct manifold learning is not really tractable as an algorithmic approach. The most powerful set of tools and theoretical basis for AI has sprung from statistical optimization theory (SGD, information-theoretical loss minimization, etc.). The fact that data is on a manifold is a tautological footnote to this approach.
crgi 1y ago
Interesting read. Reminded me of the Trinity 3D manifold visualization tool which (among other things) let's you explore the hyperspace of neural networks: https://github.com/trinity-xai/Trinity https://github.com/trinity-xai/Trinity
vayllon 1y ago
Another type of topology you’ll encounter in deep neural networks (DNNs) is network topology. This refers to the structure of the network — how the nodes are connected and how data flows between them. We already have several well-known examples, such as auto-encoders, convolutional neural networks (CNNs), and generative adversarial networks (GANs), all of which are bio-inspired. However, we still have much to learn about the topology of the brain and its functional connectivity. In the coming years, we are likely to discover new architectures — both internal within individual layers/nodes and in the ways specialized networks connect and interact with each other. Additionally, the brain doesn’t rely on a single network, but rather on several ones — often referred to as the "Big 7" — that operate in parallel and are deeply interconnected. Some of these include the Default Mode Network (DMN), the Central Executive Network (CEN) or the Limbic Network, among others. In fact, a single neuron can be part of multiple networks, each serving different functions. We have not yet been able to fully replicate this complexity in artificial systems, and there is still much to be learned and inspired by from this "network topologies". So, "Topology is all you need" :-)
mbowcut2 1y ago
To a topologist, everything is topology.
- kookamamie 1y ago
  To a man with only a hammer, everything looks like a nail.
Graviscalar 1y ago
I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.
- rubitxxx8 1y ago
  What would you have expected to happen? Advances and insights sometimes lie dormant for decades or more before someone else picks them up and does something new.
  - Graviscalar 1y ago
    I would expect model/algorithm improvements from using topological concepts to analyze the manifolds in question or concrete results in model interpretability. Gunnar has studied some toy examples, but they were barely a step up from the ones Olah constructed for the sake of explanation and they haven't borne any further fruit. You can say any advance or insight is just lying dormant, it doesn't mean anything unless you can specifically articulate why it still has potential. I haven't made any claims on the future of the intersection of deep learning and topology, I was pointing out that it's been anything but dormant given the interest in it but it hasn't lead anywhere.
- theahura 1y ago
  I think it's incorrect that the insight has gone nowhere. See, for eg, contrastive loss / clip, or vqgan image generation. Arguably also diffusion models. More generally, in my experience as an AI researcher, understandings of the geometry of data leads directly to changes in model architecture. Though people disparage that as "trial and error" it is far more directed than people on the outside give credit for.
  - Graviscalar 1y ago
    The geometric intuition is solid, but actually applying topology has been less fruitful in spite of a lot of people trying their best, as Chris Olah himself has said elsewhere in this thread.
k__ 1y ago
[flagged]
adsharma 1y ago
Isn't Deep Learning more like Graph Theory? I shared yesterday that Google published a paper called CRISP (https://arxiv.org/pdf/2505.11471 https://arxiv.org/pdf/2505.11471) that carefully avoids any reference to the word "Graph". So then the question becomes what's the difference between Graph Theory and Applied Topology? Graphs operate on discrete structures and topology is about a continuous space. Otherwise they're very closely related. But the higher order bit is that AI/ML and Deep Learning in particular could do a better job of learning from and acknowledging prior art from related fields. Reusing older terminology instead of inventing new.
kookamamie 1y ago
> This has been enough to get us to AGI.1 Hard disagree.
colah3 1y ago
Since this post is based on my 2014 blog post (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks, for example in these follow ups: - https://colah.github.io/posts/2014-10-Visualizing-MNIST/ https://colah.github.io/posts/2014-10-Visualizing-MNIST/ - https://colah.github.io/posts/2015-01-Visualizing-Representations/ https://colah.github.io/posts/2015-01-Visualizing-Representa... There are places I've found the topological perspective useful, but after a decade of grappling with trying to understand what goes on inside neural networks, I just haven't gotten that much traction out of it. I've had a lot more success with: * The linear representation hypothesis - The idea that "concepts" (features) correspond to directions in neural networks. * The idea of circuits - networks of such connected concepts. Some selected related writing: - https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/ - https://transformer-circuits.pub/2022/mech-interp-essay/index.html https://transformer-circuits.pub/2022/mech-interp-essay/inde... - https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...
- montebicyclelo 1y ago
  Related to ways of understanding neural networks, I've seen these views expressed a lot, which to me seem like misconceptions: - LLMs are basically just slightly better `n-gram` models - The idea of "just" predicting the next token, as if next-token-prediction implies a model must be dumb (I wonder if this [1] popular response to Karpathy's RNN [2] post is partly to blame for people equating language neural nets with n-gram models. The stochastic parrot paper [3] also somewhat equates LLMs and n-gram models, e.g. "although she primarily had n-gram models in mind, the conclusions remain apt and relevant". I guess there was a time where they were more equivalent, before the nets got really really good) [1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139 https://nbviewer.org/gist/yoavg/d76121dfde2618422139 [2] https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ [3] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
  - 1y ago
nis0s 1y ago
> One way to think about neural networks, especially really large neural networks, is that they are topology generators. That is, they will take in a set of data and figure out a topology where the data has certain properties. Those properties are in turn defined by the loss function. Latent spaces may or may not have useful topology, so this idea is inherently wrong, and builds the wrong type of intuition. Different neural nets will result in different feature space understanding of the same data, so I think it's incorrect to believe you're determining intrinsic geometric properties from a given neural net. I don't think people should throw around words carelessly because all that does is increase misunderstanding of concepts. In general, manifolds can help discern useful characteristics about the feature space, and may have useful topological structures, but trying to impose an idea of "topology" on this is a stretch. Moreover, the kind of basics examples used in this blog post don't help prove the author's point. Maybe I am misunderstanding this author's description of what they mean, but this idea of manifold learning is nothing new.
khoangothe 1y ago
Cool post! Thanks
deepburner 1y ago
This whole article is just a nothingburger. Saying something is applied topology is only one step more advanced than saying something is maths - duh. These mathematical abstractions are incredibly general and and you can pretty much draw up anything in terms of anything, the challenging part is being able to turn around and use the model/abstraction to say things about the thing you're abstracting. I don't think scholars have been very successful in that regard, less so this article. Yeah deep learning is applied topology, it's also applied geometry, and probably applied algebra and I wouldn't be surprised if it was also applied number theory.
ada1981 1y ago
For the last few years, I've been "seeing maps" whenever I think about LLMs. It's always felt like the most natural way to understand what is going on. It's also for this reason that I think new knowledge is discoverable from with in LLMs. I imagine having a topographic map of some island that has only been explored partially by humans. But if I know the surrounding topography, I can make pretty accurate guesses about the areas I haven't been. And I think the same thing can be applied to certain areas of human knowledge, especially when represented as text or symbolically.
revskill 1y ago
So ml is about drawing the line.
Koshkin 1y ago
On the TDA in general, I found this series of articles by Brandon Brown helpful: https://github.com/outlace/outlace.github.io https://github.com/outlace/outlace.github.io
CliffStoll 1y ago
Applied topology. Might a Klein bottle actually be a useful?
- colah3 1y ago
  It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation-features https://distill.pub/2020/circuits/equivariance/#hue-rotation... (This is, in some sense, for similar reason to Gunnar Carlson et al finding a Klein bottle when looking at high-contrast image patches, except one level more abstract, since it's about features rather than data points.)
garganzol 1y ago
I want to share one more related observation: by definition, topology math refers to geometrical objects and transformations. But there exists another, more computer-esque definition of topology that defines relations between abstract objects. For example, let's take a look at graph data structure. A graph has a set of stored objects (vertices) and a set of stored relations between the vertices (edges). In this way, graph defines a topology in discrete form. Let's take a look at network data structure which is closely related to the graph. It is very much the same idea, but it additionally has a value stored in every edge. A network has a set of objects (vertices) and a set of relations between the objects (edges), while edges also hold edge values. So it is also a form of topology because the network defines the relations between the abstract objects. In this light, you can view a graph as a neural network with {0, 1} weights. The graph edge is either present or absent, hence {0, 1} values only. The network structure, however, can hold any assigned value in every edge, so every connection between objects (neurons) can be characterized not only by its presence, but also by edge-assigned values (weights). Now we get the full model of a neural network. And yes, it is built upon topology in its discrete form.
fasdfdsa 1y ago
[flagged]
strimp099 1y ago
One of the most successful quant traders in history, Jim Simons studied topology in the 60s. It’s rumored he used deep neural networks in his trading before they were cool. This post really brought the two together for me in a way I didn’t understand before.
paraschopra 1y ago
Aren't manifolds generally task-dependent? I've been debating whether the data lies on a manifold, or whether the data attributes that are task-relevant (and of our interest) lie on a manifold? I suspect it is the latter, but I've seen Platonic Representation Hypothesis that seems to hint it is the former.
xchip 1y ago
The deep learning you are talking about is better understood as a regression.
techlatest_net 1y ago
[dead]
wg0 1y ago
This is an insightful read but at a place where a prompt churns out a whole model - that's yet far far away and extremely compute intensive. Philosophical point being - if human brain is also all planes all the way down with trillion dimensional planes separating concepts. I don't think so that is the case but yes - we might mimic something like that and such a thing will always malfunction more than their human counterparts in a given subject area and would always hit a wall where a human won't.
iFire 1y ago
Huge fan of the idea in fiction of Applied Topology is magic https://en.wikipedia.org/wiki/The_Laundry_Files https://en.wikipedia.org/wiki/The_Laundry_Files > Howard is recruited to work for the Q-Division of SOE, otherwise known as "the Laundry", the British government agency which deals with occult threats. "Magic" is described as being a branch of applied computation (mathematics), therefore computers and equations are just as useful, and perhaps more potent, than classic spellbooks, pentagrams, and sigils for the purpose of influencing ancient powers and opening gates to other dimensions.
lesostep 1y ago
>as long as we can separate good from bad we can train a neural network to sort out the topology for us. 10-ish years ago, I saw a project training networks to guess biological sex from face photos. They carefully removed makeup, moustache, hair, etc, so the model would be unbiased, yet they only reached 70 to 80% correct guesses. Yet it seemed like a great result, and they were trying to reach 99%. First thing I did after reading their paper, was seek a paper where people would try and guess the biological sex from similar photos. And people weren't that much better at it. The difference between people and machine guessing was 1 or 2 percent. I asked the guys that run the project, how they proved that such a division, based only on a photo, was even possible. They didn't understand the question, they just assumed that you can do it. They couldn't improve their results in the end. Maybe they sucked at teaching neural networks, or maybe a lot of faces just are unisex if you remove gender markers. I bring this anecdote because this guys, in my eyes, made a reasonable assumption. An assumption that since they in most situations can guess what's in someone pants by seeing someones face, the face has this information. The assumption that we could somehow separate good from bad, when we rewrite school books every year, when we try to calculate "half-life of knowledge", when philosophy as a science isn't over, and every day there are political and ideological debates about what's best, is a very-very unreasonable assumption.
- lesostep 1y ago
  I forgot to conclude: In the end, it's not even reasonable to assume that such a divide between "good" and "bad" exists at all.

508 points by theahura 1y ago

If it was topology we wouldn't bother to warp the manifold so we can do similarity search. No, it's geometry, with a metric. Just as in real life, we want to be able to compare things. Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? I imagine it violently changing at first before stabilizing, followed by geometric refinement. Here are some relevant papers: * Topology and geometry of data manifold in deep learning (https://arxiv.org/abs/2204.08624 https://arxiv.org/abs/2204.08624) * Topology of Deep Neural Networks (https://jmlr.org/papers/v21/20-345.html https://jmlr.org/papers/v21/20-345.html) * Persistent Topological Features in Large Language Models (https://arxiv.org/abs/2410.11042 https://arxiv.org/abs/2410.11042) * Deep learning as Ricci flow (https://www.nature.com/articles/s41598-024-74045-9 https://www.nature.com/articles/s41598-024-74045-9)

profchemai 1y ago
Agree, if anything it's Applied Linear Algebra...but that sounds less exotic.
- lostmsu 1y ago
  Well, we know it is non-linear. More like differential equations.
theahura 1y ago
> Topological transformation of the manifold happens during training too. That makes me wonder: how does the topology evolve during training? If you've ever played with GANs or VAEs, you can actually answer this question! And the answer is more or less 'yes'. You can look at GANs at various checkpoints during training and see how different points in the high dimensional space move around (using tools like UMAP / TSNE). > I imagine it violently changing at first before stabilizing, followed by geometric refinement Also correct, though the violent changing at the beginning is also influenced the learning rate and the choice of optimizer.
- esafak 1y ago
  And crucially, the initialization algorithm.

umutisik 1y ago

Data doesn't actually live on a manifold. It's an approximation used for thinking about data. Near total majority, if not 100%, of the useful things done in deep learning have come from not thinking about topology in any way. Deep learning is not applied anything, it's an empirical field advanced mostly by trial and error and, sure, a few intuitions coming from theory (that was not topology).

motoboi 1y ago
Your comment sits in the nice gradient between not seeing at all the obvious relationships between deep learning and topology and thinking that deep learning is applied topology. See? Everything lives in the manifold. Now for a great visualization about the Manifold Hypothesis I cannot recommend more this video: https://www.youtube.com/watch?v=pdNYw6qwuNc https://www.youtube.com/watch?v=pdNYw6qwuNc That helps to visualize how the activation functions, bias and weights (linear transformations) serve to stretch the high dimensional space so that data go into extremes and become easy to put in a high dimension, low dimensional object (the manifold) where is trivial to classify or separate. Gaining an intuition about this process will make some deep learning practices so much easy to understand.
thuuuomas 1y ago
I cannot understand this prideful resentment of theory common among self-described practitioners. Even if existing theory is inadequate, would an operating theory not be beneficial? Or is the mystique combined with guess&check drudgery job security?
- hiddencost 1y ago
  Maybe a little less with the ad hominems? The OP is providing an accurate description of an extremely immature field.
  - cnity 1y ago
    Many mathematicians are (rightly, IMO) allergic to assertions that certain branches are not useful (explicit in OP) and especially so if they are dismissive of attempts to understand complicated real world phenomema (implicit in OP, if you ask me).

mirekrusin 1y ago

Just because manifold looks a bit like burrito if you squint doesn't mean it is a burrito.

ComplexSystems 1y ago
What if you don't have to squint very much?
- mirekrusin 1y ago
  Same amount of squinting needed as for claim that deep learning is just a bunch of matrices. Or 0s and 1s. Cool. Now what?

sota_pop 1y ago

I’ve always enjoyed this framing of the subject, the idea of mapping anything as hyperplanes existing in a solution space is one of the ideas that really blew my hair back during my academic studies. I would nitpick at your “dots in a circle example - with the stoner reference joke” I could be mistaken, but common practice isn’t to “move to a higher dimension”, but use a kernel (i.e. parameterize the points into the polar |r,theta> basis). All things considered, nice article.

theahura 1y ago
I'm pulling directly from Chris Olah's blog post with that example. But I will say that in practice, its always surprising how increasing the dimensionality of a neural network magically solves all sorts of problems. You could use a kernel if you don't have more computation available, but given more computation adding a dimension is strictly more flexible (and is capable of separating a much wider range of datasets)
- benl 1y ago
  Your explanation of finding a surface to separate good reasoning traces from bad reasoning traces in a high dimensional space worked as a great framing of the problem. It seems though that the surface will be fractal - the distance between a good trace and a bad trace could be arbitrarily small. If so then the work required to find and compute better and better surfaces will grow arbitrarily large. I wonder if there is a rigorous way to determine if the surface is fractal or not.

polotics 1y ago

Ok, how do transformers fit into this understanding of deep learning?

motoboi 1y ago
Transformers don't feel differentiable (because of the attention mechanism), but they actually are (as being back-propagation based forces it to be). The attention mechanism is not a stretching of the manifold, but is trained to be able to measure distances in the manifold surface, which is stretched and deformed (or transformed?) in the feed-forward layers.
theahura 1y ago
Transformers learn embedding representations of tokens, which are easily mapped into a space. Similar tokens are mapped to similar places on the space. The fully connected layer at the end of each transformer block defines a transformation of a set of points in a space to another point in that space, not unlike the example of adding colors together to get a new color
jebarker 1y ago
Transformers (with self-attention being the key operation) are kernel smoothers which fits easily into this view of the world. See here: http://bactra.org/notebooks/nn-attention-and-transformers.html http://bactra.org/notebooks/nn-attention-and-transformers.ht...

parpfish 1y ago

This is also how I've often thought about deep learning -- focusing on the geometry of the data at each layer rather than the weights and biases is far more revealing. I've always been hopeful that some algebraic topology master would dig into this question and it'd provide some better design principles for neural nets. which activation functions? how much to fan in/out? how many layers?

amelius 1y ago

[flagged]

adverbly 1y ago
Life is just applied category theory
deleted 1y ago
[deleted]

profchemai 1y ago

Once I read "This has been enough to get us to AGI.", credibility took a nose dive. In general it's a nice idea, but the blogpost is very fluffy, especially once it connects it to reasoning, there is serious technical work in this area (i.g. https://arxiv.org/abs/1402.1869 https://arxiv.org/abs/1402.1869) that has expanded this idea and made it more concrete.

Koshkin 1y ago

[flagged]

deleted 1y ago
[deleted]

fedeb95 1y ago

Interesting read. This seems hard to prove: "Everything lives on a manifold"

ComplexSystems 1y ago

I really liked this article, though I don't know why the author is calling the idea of finding a separating surface between two classes of points "topology." For instance, they write "If you are trying to learn a translation task — say, English to Spanish, or Images to Text — your model will learn a topology where bread is close to pan, or where that picture of a cat is close to the word cat." This is everything that topology is not about: a notion of points being "close" or "far." If we have some topological space in which two points are "close," we can stretch the space so as to get the same topological space, but with the two points now "far". That's the whole point of the joke that the coffee cup and the donut are the same thing. Instead, the entire thing seems to be a real-world application of something like algebraic geometry. We want to look for something like an algebraic variety the points are near. It's all about geometry and all about metrics between points. That's what it seems like to me, anyway.

srean 1y ago
> This is everything that topology is not about 100 percent true. I can only hope that in an article that is about two things, i) topology and ii) deep learning, the evident confusions are contained within one of them -- topology, only.
theahura 1y ago
fair, I was using 'topology' more colloquially in that sentence. Should have said 'surface'.
- srean 1y ago
  Ah! That clears it up. You then mean Deep Learning has a lot in common with differential geometry and manifolds in general. That I will definitely agree with. DG and manifolds have far richer and informative structure than topology.
steppi 1y ago
If I had to give a loose definition of topology, I would say that it is actually about studying spaces which have some notion of what is close and far, even if no metric exists. The core idea of neighborhoods in point set topology captures the idea of points being nearby another point, and allows defining things like continuity and sequence convergence which require a notion of closeness. From Wikipedia [0] for example The terms 'nearby', 'arbitrarily small', and 'far apart' can all be made precise by using the concept of open sets. If we change the definition of 'open set', we change what continuous functions, compact sets, and connected sets are. Each choice of definition for 'open set' is called a topology. A set with a topology is called a topological space. Metric spaces are an important class of topological spaces where a real, non-negative distance, also called a metric, can be defined on pairs of points in the set. Having a metric simplifies many proofs, and many of the most common topological spaces are metric spaces. That's not to say that topology is necessarily the best lens for understanding neural networks, and the article's author has shown up in the comments to state he's moved on in his thinking. I'm just trying to clear up a misconception. [0] https://en.wikipedia.org/wiki/General_topology https://en.wikipedia.org/wiki/General_topology

Koshkin 1y ago

Speaking of: https://www2.math.upenn.edu/~ghrist/notes.html https://www2.math.upenn.edu/~ghrist/notes.html

deleted 1y ago

[deleted]

soulofmischief 1y ago

Thanks for sharing. I also tend to view learning in terms of manifolds. It's a powerful representation. > I'm personally pretty convinced that, in a high enough dimensional space, this is indistinguishable from reasoning I actually have journaled extensively about this and even written some on Hacker News about it with respect to what I've been calling probabilistic reasoning manifolds: > This manifold is constructed via learning a decontextualized pattern space on a given set of inputs. Given the inherent probabilistic nature of sampling, true reasoning is expressed in terms of probabilities, not axioms. It may be possible to discover axioms by locating fixed points or attractors on the manifold, but ultimately you're looking at a probabilistic manifold constructed from your input set. > But I don't think you can untie this "reasoning" from your input data. It's possible you will find "meta-reasoning", or similar structures found in any sufficiently advanced reasoning manifold, but these highly decontextualized structures might be entirely useless without proper recontextualization, necessitating that a reasoning manifold is trained on input whose patterns follow learnable underlying rules, if the manifold is to be useful for processing input of that kind. > Decontextualization is learning, decomposing aspects of an input into context-agnostic relationships. But recontextualization is the other half of that, knowing how to take highly abstract, sometimes inexpressible, context-agnostic relationships and transform them into useful analysis in novel domains Full comment: https://news.ycombinator.com/item?id=42871894 https://news.ycombinator.com/item?id=42871894

mjburgess 1y ago
Are you talking about reasoning in general, reasoning qua that mental process which operates on (representations of) propositions? In which case, I cannot understand " true reasoning is expressed in terms of probabilities, not axioms " One of the features of reasoning is that it does not operate in this way. It's highly implausible animals would have been endowed with no ability to operate non-probabilistically on propositions represented by them, since this is essential for correct reasoning -- and a relatively trivial capability to provide. Eg., "if the spider is in boxA, then it is not everywhere else" and so on
- jvanderbot 1y ago

maxiepoo 1y ago

Isn't it more differential geometry?

srean 1y ago

The title, as it stands, is trite and wrong. More about that a little later. The article on the other hand is a pleasant read. Topology is whatever little structure that remains in geometry after you throwaway distances, angles, orientations and all sorts of non tearing stretchings. It's that bare minimum that still remains valid after such violent deformations. While notion of topology is definitely useful in machine learning, -- scale, distance, angles etc., all usually provide lots of essential information about the data. If you want to distinguish between a tabby cat and a tiger it would be an act of stupidity to ignore scale. Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations. That happens, but to claim deep learning is applied topology is absurd, almost stupid.

theahura 1y ago
> Topology is useful especially when you cannot trust lengths, distances angles and arbitrary deformations But...you can't. The input data lives on a manifold that you cannot 'trust'. It doesn't mean anything apriori that an image of a coca-cola can and an image of a stopsign live close to each other in pixel space. The neural network applies all of those violent transformations you are talking about
- srean 1y ago
  > But...you can't. Only in a desperate sales pitch or a desparate research grants. There are of course some situations were certain measurements are untrustworthy, but to claim that is the common case is very snake oily. When certain measurements become untrustworthy, that it does so only because of some unknown smooth transformation, is not very frequent (this is what purely topological methods will deal with). Random noise will also do that for you. Not disputing the fact that sometimes metrics cannot be trusted entirely, but to go to a topological approach seems extreme. One should use as much of the relevant non-topological information as possible. As the hackneyed example goes a topological methods would not be able to distinguish between a cup and a donut. For that you would need to trust non-topological features such as distances and angles. Deep learning methods can indeed differentiate between cop-nip and coffee mugs. BTW I am completely on-board with the idea that data often looks as if it has been sampled from an unknown, potentially smooth, possibly non-Euclidean manifold and then corrupted by noise. In such cases recovering that manifold from noisy data is a very worthy cause. In fact that is what most of your blogpost is about. But that's differential geometry and manifolds, they have structure far richer than a topology. For example they may have tangent planes, a Reimann metric or a symplectic form etc. A topological method would throw all of that away and focus on topology.

deleted 1y ago

[deleted]

terabytest 1y ago

I'm confused by the author's diagram claiming that AGI/ASI are points on the same manifold as next token prediction, chat models, and CoT models. While the latter three are provably part of the same manifold, what justifies placing AGI/ASI there too? What if the models capable of CoT aren't and will never be, regardless of topological manipulation, capable of processes that could be considered AGI? For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing. As a layman, this matches my intuition that LLMs are not at all in the same family of systems as the ones capable of generating intelligence or consciousness.

theahura 1y ago
Possible. AGI/ASI are poorly defined. I tend to think we're already at AGI, obviously many disagree. > For example, human intelligence (the closest thing we know to AGI) requires extremely complex sensory and internal feedback loops and continuous processing unlike autoregressive models' discrete processing. I've done a fair bit of connectomics research and I think that this framing elides the ways in which neural networks and biological networks are actually quite similar. For example, in mice olfactory systems there is something akin to a 'feature vector' that appears based on which neurons light up. Specific sets of neurons lighting up means 'chocolate' or 'lemon' or whatever. More generally, it seems like neuronal representations are somewhat similar to embedding representations, and you could imagine constructing an embedding space based on what neurons light up where. Everything on top of the embeddings is 'just' processing.
- fusionadvocate 1y ago
  I believe we already have the technology required for AGI. It perhaps is analogous to a lunar manned station or a 2 mile tall skyscrapper. We have the technology required to build it, but we don't for various reasons.

_alternator_ 1y ago

The question is not so much whether this is true—we can certainly represent any data as points on a manifold. Rather, it’s the extent to which this point of view is useful. In my experience, it’s not the most powerful perspective. In short, direct manifold learning is not really tractable as an algorithmic approach. The most powerful set of tools and theoretical basis for AI has sprung from statistical optimization theory (SGD, information-theoretical loss minimization, etc.). The fact that data is on a manifold is a tautological footnote to this approach.

crgi 1y ago

Interesting read. Reminded me of the Trinity 3D manifold visualization tool which (among other things) let's you explore the hyperspace of neural networks: https://github.com/trinity-xai/Trinity https://github.com/trinity-xai/Trinity

vayllon 1y ago

Another type of topology you’ll encounter in deep neural networks (DNNs) is network topology. This refers to the structure of the network — how the nodes are connected and how data flows between them. We already have several well-known examples, such as auto-encoders, convolutional neural networks (CNNs), and generative adversarial networks (GANs), all of which are bio-inspired. However, we still have much to learn about the topology of the brain and its functional connectivity. In the coming years, we are likely to discover new architectures — both internal within individual layers/nodes and in the ways specialized networks connect and interact with each other. Additionally, the brain doesn’t rely on a single network, but rather on several ones — often referred to as the "Big 7" — that operate in parallel and are deeply interconnected. Some of these include the Default Mode Network (DMN), the Central Executive Network (CEN) or the Limbic Network, among others. In fact, a single neuron can be part of multiple networks, each serving different functions. We have not yet been able to fully replicate this complexity in artificial systems, and there is still much to be learned and inspired by from this "network topologies". So, "Topology is all you need" :-)

mbowcut2 1y ago

To a topologist, everything is topology.

kookamamie 1y ago
To a man with only a hammer, everything looks like a nail.

Graviscalar 1y ago

I was one of the people that was super excited after reading the Chris Olah blogpost from 2014, and over the past decade I've seen the insight go exactly nowhere. It's neat but it hasn't driven any interesting results, though Ayasdi did some interesting stuff with TDA and Gunnar Carlson has been playing around with neural nets recently.

rubitxxx8 1y ago
What would you have expected to happen? Advances and insights sometimes lie dormant for decades or more before someone else picks them up and does something new.
- Graviscalar 1y ago
  I would expect model/algorithm improvements from using topological concepts to analyze the manifolds in question or concrete results in model interpretability. Gunnar has studied some toy examples, but they were barely a step up from the ones Olah constructed for the sake of explanation and they haven't borne any further fruit. You can say any advance or insight is just lying dormant, it doesn't mean anything unless you can specifically articulate why it still has potential. I haven't made any claims on the future of the intersection of deep learning and topology, I was pointing out that it's been anything but dormant given the interest in it but it hasn't lead anywhere.
theahura 1y ago
I think it's incorrect that the insight has gone nowhere. See, for eg, contrastive loss / clip, or vqgan image generation. Arguably also diffusion models. More generally, in my experience as an AI researcher, understandings of the geometry of data leads directly to changes in model architecture. Though people disparage that as "trial and error" it is far more directed than people on the outside give credit for.
- Graviscalar 1y ago
  The geometric intuition is solid, but actually applying topology has been less fruitful in spite of a lot of people trying their best, as Chris Olah himself has said elsewhere in this thread.

k__ 1y ago

[flagged]

adsharma 1y ago

Isn't Deep Learning more like Graph Theory? I shared yesterday that Google published a paper called CRISP (https://arxiv.org/pdf/2505.11471 https://arxiv.org/pdf/2505.11471) that carefully avoids any reference to the word "Graph". So then the question becomes what's the difference between Graph Theory and Applied Topology? Graphs operate on discrete structures and topology is about a continuous space. Otherwise they're very closely related. But the higher order bit is that AI/ML and Deep Learning in particular could do a better job of learning from and acknowledging prior art from related fields. Reusing older terminology instead of inventing new.

kookamamie 1y ago

> This has been enough to get us to AGI.1 Hard disagree.

colah3 1y ago

Since this post is based on my 2014 blog post (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks, for example in these follow ups: - https://colah.github.io/posts/2014-10-Visualizing-MNIST/ https://colah.github.io/posts/2014-10-Visualizing-MNIST/ - https://colah.github.io/posts/2015-01-Visualizing-Representations/ https://colah.github.io/posts/2015-01-Visualizing-Representa... There are places I've found the topological perspective useful, but after a decade of grappling with trying to understand what goes on inside neural networks, I just haven't gotten that much traction out of it. I've had a lot more success with: * The linear representation hypothesis - The idea that "concepts" (features) correspond to directions in neural networks. * The idea of circuits - networks of such connected concepts. Some selected related writing: - https://distill.pub/2020/circuits/zoom-in/ https://distill.pub/2020/circuits/zoom-in/ - https://transformer-circuits.pub/2022/mech-interp-essay/index.html https://transformer-circuits.pub/2022/mech-interp-essay/inde... - https://transformer-circuits.pub/2025/attribution-graphs/biology.html https://transformer-circuits.pub/2025/attribution-graphs/bio...

montebicyclelo 1y ago
Related to ways of understanding neural networks, I've seen these views expressed a lot, which to me seem like misconceptions: - LLMs are basically just slightly better `n-gram` models - The idea of "just" predicting the next token, as if next-token-prediction implies a model must be dumb (I wonder if this [1] popular response to Karpathy's RNN [2] post is partly to blame for people equating language neural nets with n-gram models. The stochastic parrot paper [3] also somewhat equates LLMs and n-gram models, e.g. "although she primarily had n-gram models in mind, the conclusions remain apt and relevant". I guess there was a time where they were more equivalent, before the nets got really really good) [1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139 https://nbviewer.org/gist/yoavg/d76121dfde2618422139 [2] https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ [3] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
- 1y ago

nis0s 1y ago

> One way to think about neural networks, especially really large neural networks, is that they are topology generators. That is, they will take in a set of data and figure out a topology where the data has certain properties. Those properties are in turn defined by the loss function. Latent spaces may or may not have useful topology, so this idea is inherently wrong, and builds the wrong type of intuition. Different neural nets will result in different feature space understanding of the same data, so I think it's incorrect to believe you're determining intrinsic geometric properties from a given neural net. I don't think people should throw around words carelessly because all that does is increase misunderstanding of concepts. In general, manifolds can help discern useful characteristics about the feature space, and may have useful topological structures, but trying to impose an idea of "topology" on this is a stretch. Moreover, the kind of basics examples used in this blog post don't help prove the author's point. Maybe I am misunderstanding this author's description of what they mean, but this idea of manifold learning is nothing new.

khoangothe 1y ago

Cool post! Thanks

deepburner 1y ago

This whole article is just a nothingburger. Saying something is applied topology is only one step more advanced than saying something is maths - duh. These mathematical abstractions are incredibly general and and you can pretty much draw up anything in terms of anything, the challenging part is being able to turn around and use the model/abstraction to say things about the thing you're abstracting. I don't think scholars have been very successful in that regard, less so this article. Yeah deep learning is applied topology, it's also applied geometry, and probably applied algebra and I wouldn't be surprised if it was also applied number theory.

ada1981 1y ago

For the last few years, I've been "seeing maps" whenever I think about LLMs. It's always felt like the most natural way to understand what is going on. It's also for this reason that I think new knowledge is discoverable from with in LLMs. I imagine having a topographic map of some island that has only been explored partially by humans. But if I know the surrounding topography, I can make pretty accurate guesses about the areas I haven't been. And I think the same thing can be applied to certain areas of human knowledge, especially when represented as text or symbolically.

revskill 1y ago

So ml is about drawing the line.

Koshkin 1y ago

On the TDA in general, I found this series of articles by Brandon Brown helpful: https://github.com/outlace/outlace.github.io https://github.com/outlace/outlace.github.io

CliffStoll 1y ago

Applied topology. Might a Klein bottle actually be a useful?

colah3 1y ago
It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation-features https://distill.pub/2020/circuits/equivariance/#hue-rotation... (This is, in some sense, for similar reason to Gunnar Carlson et al finding a Klein bottle when looking at high-contrast image patches, except one level more abstract, since it's about features rather than data points.)

garganzol 1y ago

I want to share one more related observation: by definition, topology math refers to geometrical objects and transformations. But there exists another, more computer-esque definition of topology that defines relations between abstract objects. For example, let's take a look at graph data structure. A graph has a set of stored objects (vertices) and a set of stored relations between the vertices (edges). In this way, graph defines a topology in discrete form. Let's take a look at network data structure which is closely related to the graph. It is very much the same idea, but it additionally has a value stored in every edge. A network has a set of objects (vertices) and a set of relations between the objects (edges), while edges also hold edge values. So it is also a form of topology because the network defines the relations between the abstract objects. In this light, you can view a graph as a neural network with {0, 1} weights. The graph edge is either present or absent, hence {0, 1} values only. The network structure, however, can hold any assigned value in every edge, so every connection between objects (neurons) can be characterized not only by its presence, but also by edge-assigned values (weights). Now we get the full model of a neural network. And yes, it is built upon topology in its discrete form.

fasdfdsa 1y ago

[flagged]

strimp099 1y ago

One of the most successful quant traders in history, Jim Simons studied topology in the 60s. It’s rumored he used deep neural networks in his trading before they were cool. This post really brought the two together for me in a way I didn’t understand before.

paraschopra 1y ago

Aren't manifolds generally task-dependent? I've been debating whether the data lies on a manifold, or whether the data attributes that are task-relevant (and of our interest) lie on a manifold? I suspect it is the latter, but I've seen Platonic Representation Hypothesis that seems to hint it is the former.

xchip 1y ago

The deep learning you are talking about is better understood as a regression.

techlatest_net 1y ago

[dead]

wg0 1y ago

This is an insightful read but at a place where a prompt churns out a whole model - that's yet far far away and extremely compute intensive. Philosophical point being - if human brain is also all planes all the way down with trillion dimensional planes separating concepts. I don't think so that is the case but yes - we might mimic something like that and such a thing will always malfunction more than their human counterparts in a given subject area and would always hit a wall where a human won't.

iFire 1y ago

Huge fan of the idea in fiction of Applied Topology is magic https://en.wikipedia.org/wiki/The_Laundry_Files https://en.wikipedia.org/wiki/The_Laundry_Files > Howard is recruited to work for the Q-Division of SOE, otherwise known as "the Laundry", the British government agency which deals with occult threats. "Magic" is described as being a branch of applied computation (mathematics), therefore computers and equations are just as useful, and perhaps more potent, than classic spellbooks, pentagrams, and sigils for the purpose of influencing ancient powers and opening gates to other dimensions.

lesostep 1y ago

>as long as we can separate good from bad we can train a neural network to sort out the topology for us. 10-ish years ago, I saw a project training networks to guess biological sex from face photos. They carefully removed makeup, moustache, hair, etc, so the model would be unbiased, yet they only reached 70 to 80% correct guesses. Yet it seemed like a great result, and they were trying to reach 99%. First thing I did after reading their paper, was seek a paper where people would try and guess the biological sex from similar photos. And people weren't that much better at it. The difference between people and machine guessing was 1 or 2 percent. I asked the guys that run the project, how they proved that such a division, based only on a photo, was even possible. They didn't understand the question, they just assumed that you can do it. They couldn't improve their results in the end. Maybe they sucked at teaching neural networks, or maybe a lot of faces just are unisex if you remove gender markers. I bring this anecdote because this guys, in my eyes, made a reasonable assumption. An assumption that since they in most situations can guess what's in someone pants by seeing someones face, the face has this information. The assumption that we could somehow separate good from bad, when we rewrite school books every year, when we try to calculate "half-life of knowledge", when philosophy as a science isn't over, and every day there are political and ideological debates about what's best, is a very-very unreasonable assumption.

lesostep 1y ago
I forgot to conclude: In the end, it's not even reasonable to assume that such a divide between "good" and "bad" exists at all.