4 ms·

>Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. I mean that's easy lol. People don't like to invest in th

>Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. I mean that's easy lol. People don't like to invest in thin air, which is what you get when you look at non-LLM alternatives to General Intelligence. This isn't meant as a jab or snide remark or anything like that. There's literally nothing else that will get you GPT-2 level performance, never-mind an IMO Gold Medalist. Invest in what else exactly? People are putting their eggs in one basket because it's the only basket that exists. >I think the big question is if/when investors will start giving money to those who have been predicting this (with evidence) and trying other avenues. Because those people have still not been proven right. Does "It's an incremental improvement over the model we released a few months ago, and blows away the model we released 2 years ago." really scream, "See!, those people were wrong all along!" to you ?

godelski 1y ago
> which is what you get when you look at non-LLM alternatives to General Intelligence. I disagree with this. There are a good ideas that are worth pursuit. I'll give you that few, if any, have been shown to work at scale but I'd say that's a self-fulfilling prophecy. If your bar is that they have to be proven at scale then your bar is that to get investment you'd have to have enough money to not need investment. How do you compete if you're never given the opportunity to compete? You could be the greatest quarterback in the world but if no one will let you play in the NFL then how can you prove that? On the other hand, investing in these alternatives is a lot cheaper, since you can work your way to scale and see what fails along the way. This is more like letting people try their stuff out in lower leagues. The problem is there's no ladder to climb after a certain point. If you can't fly then how do you get higher? > Invest in what else exactly? ... it's the only basket that exists. I assume you don't work in ML research? I mean that's okay but I'd suspect that this claim would come from someone not on the inside. Though tbf, there's a lot of ML research that is higher level and not working on alternative architectures. I guess the two most well known are Mamba and Flows. I think those would be known by the general HN crowd. While I think neither will get us to AGI I think both have advantages that shouldn't be ignored. Hell, even scaling a very naive Normalizing Flow (related to Flow Matching) has been shown to compete and beat top diffusion models[0,1]. The architectures aren't super novel here but they do represent the first time a NF was trained above 200M params. That's a laughable number by today's standards. I can even tell you from experience that there's a self-fulfilling filtering for this kind of stuff because having submitted works in this domain I'm always asked to compare with models >10x my size. Even if I beat them on some datasets people will still point to the larger model as if that's a fair comparison (as if a benchmark is all the matters and doesn't need be contextualized). > Because those people have still not been proven right. You're right. But here's the thing. *NO ONE HAS BEEN PROVEN RIGHT*. That condition will not exist until we get AGI. > scream, "See!, those people were wrong all along!" to you ? Let me ask you this. Suppose people are saying "x is wrong, I think we should do y instead" but you don't get funding because x is currently leading. Then a few years later y is proven to be the better way of doing things, everything shifts that way. Do you think the people who said y was right get funding or do you think people who were doing x but then just switched to y after the fact get funding? We have a lot of history to tell us the most common answer... [0] https://arxiv.org/abs/2412.06329 https://arxiv.org/abs/2412.06329 [1] https://arxiv.org/abs/2506.06276 https://arxiv.org/abs/2506.06276
- famouswaffles 1y ago
  >I disagree with this. There are a good ideas that are worth pursuit. I'll give you that few, if any, have been shown to work at scale but I'd say that's a self-fulfilling prophecy. If your bar is that they have to be proven at scale then your bar is that to get investment you'd have to have enough money to not need investment. How do you compete if you're never given the opportunity to compete? You could be the greatest quarterback in the world but if no one will let you play in the NFL then how can you prove that? On the other hand, investing in these alternatives is a lot cheaper, since you can work your way to scale and see what fails along the way. This is more like letting people try their stuff out in lower leagues. The problem is there's no ladder to climb after a certain point. If you can't fly then how do you get higher? I mean this is why I moved the bar down from state of the art. I'm not saying there are no good ideas. I'm saying none of them have yet shown enough promise to be called another basket in it's own right. Open AI did it first because they really believed in scaling, but anyone (well not literally, but you get what I mean) could have trained GPT-2. You didn't need some great investment, even then. It's that level of promise I'm saying doesn't even exist yet. >I guess the two most well known are Mamba and Flows. I mean, Mamba is a LLM ? In my opinion, it's the same basket. I'm not saying it has to be a transformer or that you can't look for ways to improve the architecture. It's not like Open AI or Deepmind aren't pursuing such things. Some of the most promising tweaks/improvements - Byte Latent Transformer, Titans etc are from those top labs. Flows research is intriguing but it's not another basket in the sense that it's not an alternative to the 'AGI' these people are trying to build. > Let me ask you this. Suppose people are saying "x is wrong, I think we should do y instead" but you don't get funding because x is currently leading. Then a few years later y is proven to be the better way of doing things, everything shifts that way. Do you think the people who said y was right get funding or do you think people who were doing x but then just switched to y after the fact get funding? We have a lot of history to tell us the most common answer... The funding will go to players positioned to take advantage. If x was leading for years then there was merit in doing it, even if a better approach came along. Think about it this way, Open AI now have 700M Weekly active users for ChatGPT and millions of API devs. If this superior y suddenly came along and materialized and they assured you there were pivoting, why wouldn't you invest in them over players starting from 0, even if they championed y in the first place? They're better positioned to give you a better return on your money. Of course, you can just invest in both. Open AI didn't get nearly a billion weekly active users off the promise of future technology. They got it with products that exist here and now. Even if there's some wall, this is clearly a road with a lot of merit. The value they've already generated (a whole lot) won't disappear if LLMs don't reach the heights some people are hoping they will. If you want people to invest in y instead then x has to stall or y has to show enough promise. It didn't take transformers many years to embed themselves everywhere because they showed a great deal of promise right from the beginning. It shouldn't be surprising if people aren't rushing to put money in y when neither has happened yet.

godelski 1y ago

> which is what you get when you look at non-LLM alternatives to General Intelligence. I disagree with this. There are a good ideas that are worth pursuit. I'll give you that few, if any, have been shown to work at scale but I'd say that's a self-fulfilling prophecy. If your bar is that they have to be proven at scale then your bar is that to get investment you'd have to have enough money to not need investment. How do you compete if you're never given the opportunity to compete? You could be the greatest quarterback in the world but if no one will let you play in the NFL then how can you prove that? On the other hand, investing in these alternatives is a lot cheaper, since you can work your way to scale and see what fails along the way. This is more like letting people try their stuff out in lower leagues. The problem is there's no ladder to climb after a certain point. If you can't fly then how do you get higher? > Invest in what else exactly? ... it's the only basket that exists. I assume you don't work in ML research? I mean that's okay but I'd suspect that this claim would come from someone not on the inside. Though tbf, there's a lot of ML research that is higher level and not working on alternative architectures. I guess the two most well known are Mamba and Flows. I think those would be known by the general HN crowd. While I think neither will get us to AGI I think both have advantages that shouldn't be ignored. Hell, even scaling a very naive Normalizing Flow (related to Flow Matching) has been shown to compete and beat top diffusion models[0,1]. The architectures aren't super novel here but they do represent the first time a NF was trained above 200M params. That's a laughable number by today's standards. I can even tell you from experience that there's a self-fulfilling filtering for this kind of stuff because having submitted works in this domain I'm always asked to compare with models >10x my size. Even if I beat them on some datasets people will still point to the larger model as if that's a fair comparison (as if a benchmark is all the matters and doesn't need be contextualized). > Because those people have still not been proven right. You're right. But here's the thing. *NO ONE HAS BEEN PROVEN RIGHT*. That condition will not exist until we get AGI. > scream, "See!, those people were wrong all along!" to you ? Let me ask you this. Suppose people are saying "x is wrong, I think we should do y instead" but you don't get funding because x is currently leading. Then a few years later y is proven to be the better way of doing things, everything shifts that way. Do you think the people who said y was right get funding or do you think people who were doing x but then just switched to y after the fact get funding? We have a lot of history to tell us the most common answer... [0] https://arxiv.org/abs/2412.06329 https://arxiv.org/abs/2412.06329 [1] https://arxiv.org/abs/2506.06276 https://arxiv.org/abs/2506.06276

famouswaffles 1y ago
>I disagree with this. There are a good ideas that are worth pursuit. I'll give you that few, if any, have been shown to work at scale but I'd say that's a self-fulfilling prophecy. If your bar is that they have to be proven at scale then your bar is that to get investment you'd have to have enough money to not need investment. How do you compete if you're never given the opportunity to compete? You could be the greatest quarterback in the world but if no one will let you play in the NFL then how can you prove that? On the other hand, investing in these alternatives is a lot cheaper, since you can work your way to scale and see what fails along the way. This is more like letting people try their stuff out in lower leagues. The problem is there's no ladder to climb after a certain point. If you can't fly then how do you get higher? I mean this is why I moved the bar down from state of the art. I'm not saying there are no good ideas. I'm saying none of them have yet shown enough promise to be called another basket in it's own right. Open AI did it first because they really believed in scaling, but anyone (well not literally, but you get what I mean) could have trained GPT-2. You didn't need some great investment, even then. It's that level of promise I'm saying doesn't even exist yet. >I guess the two most well known are Mamba and Flows. I mean, Mamba is a LLM ? In my opinion, it's the same basket. I'm not saying it has to be a transformer or that you can't look for ways to improve the architecture. It's not like Open AI or Deepmind aren't pursuing such things. Some of the most promising tweaks/improvements - Byte Latent Transformer, Titans etc are from those top labs. Flows research is intriguing but it's not another basket in the sense that it's not an alternative to the 'AGI' these people are trying to build. > Let me ask you this. Suppose people are saying "x is wrong, I think we should do y instead" but you don't get funding because x is currently leading. Then a few years later y is proven to be the better way of doing things, everything shifts that way. Do you think the people who said y was right get funding or do you think people who were doing x but then just switched to y after the fact get funding? We have a lot of history to tell us the most common answer... The funding will go to players positioned to take advantage. If x was leading for years then there was merit in doing it, even if a better approach came along. Think about it this way, Open AI now have 700M Weekly active users for ChatGPT and millions of API devs. If this superior y suddenly came along and materialized and they assured you there were pivoting, why wouldn't you invest in them over players starting from 0, even if they championed y in the first place? They're better positioned to give you a better return on your money. Of course, you can just invest in both. Open AI didn't get nearly a billion weekly active users off the promise of future technology. They got it with products that exist here and now. Even if there's some wall, this is clearly a road with a lot of merit. The value they've already generated (a whole lot) won't disappear if LLMs don't reach the heights some people are hoping they will. If you want people to invest in y instead then x has to stall or y has to show enough promise. It didn't take transformers many years to embed themselves everywhere because they showed a great deal of promise right from the beginning. It shouldn't be surprising if people aren't rushing to put money in y when neither has happened yet.