4 ms·
Modular Manifolds
- jasonjmcghee 1y agoThe learning rates they demonstrate are crazy - though the standard when talking about CIFAR-10 is 94% accuracy iirc. Showing ~60% accuracy is weird. Has DAWNBench been done with manifold Muon (with a more appropriate architecture)?
- snake_doc 1y agoUm.. the model is tiny: https://github.com/thinking-machines-lab/manifolds/blob/main/src/main.py https://github.com/thinking-machines-lab/manifolds/blob/main...
- jasonjmcghee 1y agoYeah, it's just the wrong architecture for the job, so I found it to be a strange example. Here's the top model on DAWNBench - https://github.com/apple/ml-cifar-10-faster/blob/main/fast_cifar_10_distributed.py https://github.com/apple/ml-cifar-10-faster/blob/main/fast_c... Trains for 15 epochs and it, like all the others is a 9 layer resnet.
- srean 1y agoUsually there's more to a ML, data-science idea (that's not a full fledged fledged out journal paper) than beating a SOTA benchmark. In fact beating SOTA is often the least interesting part of an interesting paper and the SOTA-blind reviewers often use it as a gatekeeping device.
- jasonjmcghee 1y agoSure, of course. Wasn't suggesting "are you beating a sota benchmark"? I'm floating the idea of an ablation that matches a realistic scenario for the dataset / task. Personally curious how manifold muon performs compared to AdamW in a throughly explored context. This is the first time I've seen a 3-layer mlp on cifar-10. I probably should have made the 9-layer ResNet part more, front-and-center / central to my point.
- srean 1y agoGot you, this time.
- Jackson__ 1y agoThey say they train for ~3 epochs. Could it be that's just not long enough of a training run? I have no idea how many epochs are usually used in those models.
- pooooooooooooop 1y agoits a 3-layer MLP as stated in the article
- snake_doc 1y agoHmmm… http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- whimsicalism 1y agothis is a bad example to claim the bitter lesson applies to, it’s about the fundamentals of optimization techniques not about tying to hand-crafted things for the solution space.
- snake_doc 1y agoAren’t they all optimization techniques at the end of the day? Now you’re just debating semantics
- whimsicalism 1y agobelieve what you want, i guess
- deleted 1y ago[deleted]
- ACCount37 1y agoDoesn't apply as long as the improvements obtained there scale with compute. Now, are there actual meaningful improvements to obtain, and do they stick around all the way to frontier runs? Unclear, really. So far, it looks like opening a can of hyperparameters.
- TimorousBestie 1y agoReminiscing about an old HN comment arguing that differential geometry was irrelevant to machine learning with a smile on my face. Happy to see this opinion expressed here, too. The more math skeptics there are out there, the longer I get to keep my job. :)
- deviation 1y agoThe world is full of useful shapes! No reason that math shouldn't :)
- srean 1y ago"I have never had to do integrate the "arctan" function by hand in my entire career" arguments are not worth engaging with. If people are happy with a job or a role that does not need math that' fine. Familiarity with Maths let's you to rise to the occasion, to become more than a replaceable cog. The thing is, unless you are trained in math you wouldn't even recognise the opportunity, that a certain kind Of Math could have been used here. In fact, even if you are trained in Math you may not see it till much later -- it needs a special eye and something in that moment. Polyhedrons were looked at for centuries after centuries by top-notch mathematicians. All missed Euler's formula, except perhaps Descartes. Often what happens is some nontrivial branch of mathematics suddenly finds a novel and impactful application. Then crowds jump in to learn that Math. But it's mostly already a little too late for them, they have missed this bus. The best case is one already knows the Math beforehand and you don't know which part will be handy. It helps if you love the subject and can afford to invest time to learn it for the love of the subject. Once in a while you happen to find yourself in the right place and the right time and with the right tools you need.
- gowld 1y ago> Often what happens is some nontrivial branch of mathematics suddenly finds a novel and impactful application. Then crowds jump in to learn that Math. But it's mostly already a little too late for them, they have missed this bus. However, in the meantime, the experts in that math have "missed the bus" on whatever the application area is, that the math expert knows not enough about because they were studying math instead.
- esafak 1y ago> This post covers one appealing way to constrain the weight matrices of a neural network—by keeping the tensors constrained to submanifolds at each layer. This opens the door to re-thinking optimization, as we can co-design optimization algorithms with these manifold constraints. As an example, we propose a manifold version of the Muon optimizer whose weights are constrained to the Stiefel manifold: the manifold of matrices with unit condition number. We conclude the post by defining the idea of a modular manifold, which is a composable manifold that attempts to make it easier to scale up and train large networks. Very good presentation. Projected gradient methods were popular during the convex optimization craze two decades ago. The ideas advanced here have precedent and seem sensible to me. My concern is whether it helps much. The test accuracy in figure 6b shows a marginal increase, and a gentler transition to the overfitting regime, suggesting the regularization is working. The higher LR did not translate to a speed up: "Manifold Muon increased the wall clock time per step compared to AdamW..." More fundamentally, I am a bit skeptical that low test accuracy is the right goal in LLMs because statistical learning theory does not adequately model the macro-behavior of very large models.
- namibj 1y ago> The test accuracy in figure 6b shows a marginal increase, and a gentler transition to the overfitting regime, suggesting the regularization is working. Sounds like it might help for online RL training regimes as those are naturally quite vulnerable to overfitting .
- jpt4 1y ago\> statistical learning theory does not adequately model the macro-behavior of very large models. Might you please elaborate on this? I recognize that "artificial neural networks are lossy de/compression algorithms" does not enumerate the nuances of these structures, but am curious whether anything in particular is both interesting and missing from SLT.
- esafak 1y agoSLT typically uses empirical risk minimization, leading to the bias-variance decomposition and a unimodal extremum as the monotonically decreasing bias supposedly balances against the monotonically increasing variance. We now know this does not accurately model overparameterized models, which exhibit double descent, and other phenomena like grokking. To explain them you have to look past classical statistics to statistical mechanics.
- uoaei 1y agoThis is exactly the kind of out-of-the-box thinking that will get us past some of the limitations of the current crop of AI architectures. Bravo to the authors.
- SubiculumCode 1y agoCurious why the authors chose the blog format over a research report?
- almostgotcaught 1y agoyou mean a paper? because it's not paper quality content?
- sudohalt 1y agoExactly, it’s like they’re targeting people who don’t really know much about ML but are easily wowed by fancy math jargon and nice drawings.
- SubiculumCode 1y agoWhich was my round about way of asking :)
- pooooooooooooop 1y agothinkingmachines likes to flex
- fmap 1y agoIsn't this an old idea? E.g., here's a textbook on optimization algorithms for matrix manifolds https://press.princeton.edu/absil https://press.princeton.edu/absil and here's a library that implements this in python for the Stiefel manifold that's the subject of this blog post: https://pymanopt.org/docs/stable/manifolds.html#module-pymanopt.manifolds.stiefel https://pymanopt.org/docs/stable/manifolds.html#module-pyman... What is novel about the approach in the blog post? Serious question, I really can't tell after reading the post.
- cs702 1y agoI don't think it's been tried at scale, with large models. It remains to be seen if it works better than conventional training schemes.
- godelski 1y ago> Isn't this an old idea? So are neural networks. So is attention. What's your point? Sometimes things need to be retried. Sometimes there are small subtle details that make or break an idea. So what's the point of acting dismissively? If an old idea that didn't work now works, then what's the problem? Besides, progress is typically iterative, not through leaps and bounds. The vast majority of things that look like leaps are just because we don't see the steps between. The reason I'm saying this is because that sentiment is often used to pass over working solutions and slows down their progress. So even if unintentional it should cause us to rethink how we respond. Otherwise we end up with such silly claims like "Einstein just used Tensors" and "Nash just used topology". In some sense these are accurate, but they are too high level descriptions (and these are real dismissals. Which again, so what? If it works, it works?). Why is "novelty" even a question? Novelty is only ever in the eyes of the beholder. > What is novel about the approach in the blog post? Serious question, I really can't tell after reading the post. Honestly, I do not know, but I'll give you my best read on it. 1) Scale: Don't underestimate the importance of this. While I don't think scale is all you need, it certainly is a critical factor. 2) Different optimization: I may be missing something, but it looks like they are using a different optimizer. They mention that they're using the muon optimizer constraining to a Stiefel manifold. Neither of those things are unique on their own, but is their combination? This is where I'm uncertain because such a thing would be easy to miss. Maybe someone did and was unsuccessful with it. Maybe someone did, but was not at scale. Maybe someone did, it worked, and just nobody noticed (that happens a lot!). So I think this is quite similar to how 99% of progress and breakthroughs are made: putting together ideas that seem unrelated and inventing some glue to generalize the process. At a high level this always looks like you're just putting existing things together, but that glue is really hard to make. And to continue that analogy, if we do a good enough job gluing things together then to anyone but an expert it'll look like there is no glue. It can be surprisingly difficult to tell if something is glued, welded, mated, milled, printed, or whatever. It usually takes a very keen eye to determine the answer non-destructively.
- aanet 1y agoNot here to comment on the _content_ of the blog post... Just wanted to say the blog post design looks super nice. Beautifully laid out, very readable typography, clear graphics, approachable design with a welcoming UX, footnotes in the side, etc. Anybody know how this is designed / styled? (I can see three.js being used, along with katex.js - but don't know more details) Thanks
- ddellacosta 1y agoUX on the other hand...I hate it when sites hijack my key commands for moving backwards and forwards in my browser history. Please don't do this!
- manas96 1y agoI think the diagrams look very similar to what Keenan Crane uses in his papers, perhaps they used that tool. I think his students have now fleshed it out for general use.
- spyder 1y agoFor me it's horrible, some scripts makes the scroll very choppy, unusable... had to disable scripts just to be able to normally scroll :-(
- cs702 1y agoTL;DR: The OP notes that we currently use all sorts of tricks of the trade, including applying normalization layers, to keep unit values in DNNs from getting too large or too small when we train them. Keeping unit values from getting too large or small prevents numerical underflow/overflow, and also helps speed up learning by keeping the magnitudes of updates small in relation to weights. The OP proposes that we should constrain weights to be in sub-manifolds with unit condition number[a] at each layer, and that we should modify/design SGD algorithms to work well within those manifolds. I find the idea compelling, but it's too early to know if it will work well at scale, you know, with large models, in the real world. -- [a] https://en.wikipedia.org/wiki/Condition_number https://en.wikipedia.org/wiki/Condition_number -- EDIT: On the other hand, yesterday I saw a paper about doing basically the opposite, letting unit values in DNNs get as big or small as they need to get... by mapping them to complex logarithms and keeping them in that domain: https://openreview.net/forum?id=SUuzb0SOGu https://openreview.net/forum?id=SUuzb0SOGu . I also found this opposing idea oddly compelling, but I don't know how well it works either, because it hasn't been tested at scale.
- robots0only 1y agoso their way to differentiate against frontier labs is to try writing research blog posts (not papers). It will be interesting to see how this plays out. I don't think that anyone serious about developing frontier models would be putting anything useful out there for others. We already see this with all the incumbents -- Google, OAI, Anthropic, xAI, DeepSeek and other chinese labs.
- qf_community 1y ago[dead]
- aghilmort 1y agoInteresting. Modular manifolds are precisely what hypertokens use for prompt compiling. Specifically, we linearize the emergent KVQ operations of an arbitrary prompt in any arbitrary model by way of interleaving error-correcting code (ECC). ECC tokens are out-of-band tokens, e.g., Unicode's Private Use Area (PUA), interleaved with raw context tokens. This construction induces an in-context associate memory. Any sort of interleaved labeling basis, e.g., A1, quick brown fox, A2, jumped lazy dog, induces a similar effect to for chaining recall & reasoning more reliably. This trick works because PUA tokens are generally untrained hence their initial embedding is still random Gaussian w.h.p. Similar effects can be achieved by simply using token combos unlikely to exist and are often in practice more effective since PUA tokens like emojis or Mandarin characters are often 2,3, or 4 tokens after tokenization vs. codeword combos like zy-qu-qwerty every k content tokens, where can be variable. Building attention architecture using modular manifolds in white / gray-box models like this new work shows vs. prompt-based black box injection is a natural next step, and so can at least anecdotally validate what they're building ahead of next paper or two. Which is all to say, absolutely great to see others building in this way!
- snake_doc 1y agoWot? Is this what AI generated non-sense has come to? This is totally unrelated.
- aghilmort 1y agoNope. Construction induces ECC-driven emergent modular manifolds in latent space during KVQ maths. Can't use any ole ECC / crux why works. More in another reply.
- glowcoil 1y agoThe original article discusses techniques for constraining the weights of a neural network to a submanifold of weight space during training. Your comment discusses interleaving the tokens of an LLM prompt with Unicode PUA code points. These are two almost completely unrelated things, so it is very confusing to me that you are confidently asserting that they are the same thing. Can you please elaborate on why you think there is any connection at all between your comment and the original article?
- yodon 1y agoIs the original Thinking Machines trademark[0] no longer active? They were the original AI company, back when AI was a completely different thing than it is today. [0] https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporation https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporatio...
- tintor 1y agoThat company is defunct since 1994; 31 years ago
- gowld 1y agoWhat does this mean?
- phoenicyan 1y agoWell-done post, I'd like to read more of their work and it's exciting to see these new ideas. Though as other people have said, the one set of empirical results that they present is a bit... confusing? I'd think they'd have some more compelling examples to present given all the pretty math. Their modular norm paper (https://arxiv.org/abs/2405.14813 https://arxiv.org/abs/2405.14813) has several more examples; see their appendix D in particular, but these are also mystifying. Yes they're interested in how things scale but am I the only one to whom it seems that the training losses they report are just not competitive with things that are currently being used?
- nenenejej 1y agohttps://archive.is/bP3BG https://archive.is/bP3BG If you like to scroll on mobile :)
- nenenejej 1y agoNice! Posts like this make me remorseful of not following a mathematics career. I'm sure some of the notation is basic (as in undergrad) but I'd need an entire weekend to understand the post.