5 ms·

Can someone explain the economics behind this ? The claim is something than will replace the transformer, a technology powering a good chunk of AI companies.

Can someone explain the economics behind this ? The claim is something than will replace the transformer, a technology powering a good chunk of AI companies. The paper's authors seems to be either from a public university, or Sepp Hochreiter's private company or labs nx-ai.com https://www.nx-ai.com/en/xlstm https://www.nx-ai.com/en/xlstm Where is the code ? What is the license ? How are they earning money ? Why publish their secret recipe ? Will they not be replicated ? How will the rewards be commensurate with the value their algorithm bring ? Who will get money from this new technology ?

imjonse 2y ago
Should all arxiv papers be backed by economic considerations or business plans?
- jampekka 2y ago
  Or any?
- AIsore 2y ago
  Nope, they should not. It is academia after all. How would you even do that in, say, pure mathematics? Concretely, I would love to know what the business plan/economic consideration of Gower's 1998 proof of Szemeredi's theorem using higher order Fourier analysis would even look like.
  - queuebert 2y ago
    Coming soon to an HFT firm near you ...
- Der_Einzige 2y ago
  Yes they should. Academia and peer review is so corrupt, gamified, and poor quality that I’d literally trust capitalist parasites more than the current regime of “publish or perish” and citation cartels. At least capitalists have something to fight over that’s worth fighting for (money). Academics will bitterly fight over the dumbest, least important shit. There’s a law about how the less something matters, the more political the fights over it will be.
  - AIsore 2y ago
    I am certainly not going to defend peer review and its inherent flaws. I am also not sure "capitalists" or the market is always as efficient as one might hope or think. But that aside, to my point above, if capitalists were to optimize "money" as you say, how would that fix publishing? Firstly, how would they ascribe a monetary value to Gower's 1998 and his other few papers that catapulted him to the Fields Medal? Are you saying these subject do not matter because no one is bidding for these papers? I fear we would not have published Heisenberg's early papers or the discovery of Penicillin if so. And over what horizon would "capitalists" optimize that monetary value (internal IRR)? Governments usually have to step in for long term IRR projects (e.g. the internet protocol's development was famously funded by DARPA and they keeping "deep learning" alive during the downturns as no one believed in short term returns ...). The UK water system or quite a few train services around the world bear witness to the fact that even in a "capitalist" society, some long term common benefits are hard to fund with short term IRR considerations even pension funds consider reasonable. Taking that observation to its perverse conclusion, if you believe in "capitalists" then you could argue that the current imperfect review system is a side effect of capitalist societies' long term research funding plan (universities, research grants, tax breaks for endowments, student grants, ...). I just think knowledge sharing is not always compatible with financial interests. And the former, to me, is the public good that academia should attain. But you get no argument from me that peer review is broken. I struggle to think, though, of a better system and doubt "money" is it, tbh.
refulgentis 2y ago
Are you asking how academics make money from giving away knowledge in papers? It's complicated
- GistNoesis 2y ago
  I don't understand at all what the monetary value of this algorithm should be. The authors are positioning themselves as a company and not merely academics : A Sepp Hochreiter's video from 6 months ago hyping xLSTM : https://youtu.be/hwIt7ezy6t8?feature=shared&t=561 https://youtu.be/hwIt7ezy6t8?feature=shared&t=561 in which he state his intent to raise €300M to make a european alternative to openai's GPT for niche domains thanks to this new method that will allow to train for cheaper and better. He recently received (2023) € 35,000 in prize money at the 5th annual German AI Award. https://www.jku.at/en/festival-university/media/detail/news/deutscher-ki-innovationspreis-fuer-jku-forscher-sepp-hochreiter/ https://www.jku.at/en/festival-university/media/detail/news/... Or is it just an academic tactic to get more funding ? To extract more work from PhD students by making them think they are going to strike it big ? How are they intending to build a moat if they publish their papers ? Will this technology be encumbered by patents/license ?
  - AIsore 2y ago
    If you are asking that question, I guess you must have wondered about this for years, right, in fact nearly a decade? I mean why would Google have bought DeepMind with them publishing in peer reviewed journals for years after? Same for Meta (formerly facebook)? I think there is a well trodden path being followed here ... and I am surprised by your surprise.
    - GistNoesis 2y ago
      Acquisitions, like for DeepMind is usually a way to hire talent. It can make sense when the technology is new and getting a few year of lead time on what is going to be a growing market may make some financial sense. In this specific xLSTM case, the industry has matured, they are just one among many (Mamba, S3Ms, transformers-variants... ), they have already been sitting on it for at least 6 months, I don't see what their play is. An other case study that's probably interesting, are the authors of the Adam Paper, https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 , (Awarded "2020: The Adam optimization paper is the world's #1 most cited scientific paper of the past five years"). Probably a few (10?,100?) billions worth value created. You can find the authors bios http://dpkingma.com/ http://dpkingma.com/ https://jimmylba.github.io/ https://jimmylba.github.io/ I think there is a huge problem with the capture and sharing of value in the whole deep-learning industry. Academia's naivety plays a role in it, Generational Shift technologies are badly rewarded. Incremental Shift technologies aren't rewarded at all. Powerful technologies into many hands with low rewards for their creators while the value they generate keeps going to the same pockets. That's a recipe for disaster. Will be a fun thing to come back in a few years to see how it had unfold.
brookst 2y ago
I think you’re making a satirical point about how commercial R&D has far outstripped academia, but it’s not 100% clear.

imjonse 2y ago

Should all arxiv papers be backed by economic considerations or business plans?

jampekka 2y ago
Or any?
AIsore 2y ago
Nope, they should not. It is academia after all. How would you even do that in, say, pure mathematics? Concretely, I would love to know what the business plan/economic consideration of Gower's 1998 proof of Szemeredi's theorem using higher order Fourier analysis would even look like.
- queuebert 2y ago
  Coming soon to an HFT firm near you ...
Der_Einzige 2y ago
Yes they should. Academia and peer review is so corrupt, gamified, and poor quality that I’d literally trust capitalist parasites more than the current regime of “publish or perish” and citation cartels. At least capitalists have something to fight over that’s worth fighting for (money). Academics will bitterly fight over the dumbest, least important shit. There’s a law about how the less something matters, the more political the fights over it will be.
- AIsore 2y ago
  I am certainly not going to defend peer review and its inherent flaws. I am also not sure "capitalists" or the market is always as efficient as one might hope or think. But that aside, to my point above, if capitalists were to optimize "money" as you say, how would that fix publishing? Firstly, how would they ascribe a monetary value to Gower's 1998 and his other few papers that catapulted him to the Fields Medal? Are you saying these subject do not matter because no one is bidding for these papers? I fear we would not have published Heisenberg's early papers or the discovery of Penicillin if so. And over what horizon would "capitalists" optimize that monetary value (internal IRR)? Governments usually have to step in for long term IRR projects (e.g. the internet protocol's development was famously funded by DARPA and they keeping "deep learning" alive during the downturns as no one believed in short term returns ...). The UK water system or quite a few train services around the world bear witness to the fact that even in a "capitalist" society, some long term common benefits are hard to fund with short term IRR considerations even pension funds consider reasonable. Taking that observation to its perverse conclusion, if you believe in "capitalists" then you could argue that the current imperfect review system is a side effect of capitalist societies' long term research funding plan (universities, research grants, tax breaks for endowments, student grants, ...). I just think knowledge sharing is not always compatible with financial interests. And the former, to me, is the public good that academia should attain. But you get no argument from me that peer review is broken. I struggle to think, though, of a better system and doubt "money" is it, tbh.

refulgentis 2y ago

Are you asking how academics make money from giving away knowledge in papers? It's complicated

GistNoesis 2y ago
I don't understand at all what the monetary value of this algorithm should be. The authors are positioning themselves as a company and not merely academics : A Sepp Hochreiter's video from 6 months ago hyping xLSTM : https://youtu.be/hwIt7ezy6t8?feature=shared&t=561 https://youtu.be/hwIt7ezy6t8?feature=shared&t=561 in which he state his intent to raise €300M to make a european alternative to openai's GPT for niche domains thanks to this new method that will allow to train for cheaper and better. He recently received (2023) € 35,000 in prize money at the 5th annual German AI Award. https://www.jku.at/en/festival-university/media/detail/news/deutscher-ki-innovationspreis-fuer-jku-forscher-sepp-hochreiter/ https://www.jku.at/en/festival-university/media/detail/news/... Or is it just an academic tactic to get more funding ? To extract more work from PhD students by making them think they are going to strike it big ? How are they intending to build a moat if they publish their papers ? Will this technology be encumbered by patents/license ?
- AIsore 2y ago
  If you are asking that question, I guess you must have wondered about this for years, right, in fact nearly a decade? I mean why would Google have bought DeepMind with them publishing in peer reviewed journals for years after? Same for Meta (formerly facebook)? I think there is a well trodden path being followed here ... and I am surprised by your surprise.
  - GistNoesis 2y ago
    Acquisitions, like for DeepMind is usually a way to hire talent. It can make sense when the technology is new and getting a few year of lead time on what is going to be a growing market may make some financial sense. In this specific xLSTM case, the industry has matured, they are just one among many (Mamba, S3Ms, transformers-variants... ), they have already been sitting on it for at least 6 months, I don't see what their play is. An other case study that's probably interesting, are the authors of the Adam Paper, https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 , (Awarded "2020: The Adam optimization paper is the world's #1 most cited scientific paper of the past five years"). Probably a few (10?,100?) billions worth value created. You can find the authors bios http://dpkingma.com/ http://dpkingma.com/ https://jimmylba.github.io/ https://jimmylba.github.io/ I think there is a huge problem with the capture and sharing of value in the whole deep-learning industry. Academia's naivety plays a role in it, Generational Shift technologies are badly rewarded. Incremental Shift technologies aren't rewarded at all. Powerful technologies into many hands with low rewards for their creators while the value they generate keeps going to the same pockets. That's a recipe for disaster. Will be a fun thing to come back in a few years to see how it had unfold.

brookst 2y ago

I think you’re making a satirical point about how commercial R&D has far outstripped academia, but it’s not 100% clear.