3 ms·

i intereviewed Jon (lead author on this paper) and yeah he pretty much disowns it now https://www.latent.space/p/mosaic-mpt-7b https://www.latent.space/p/mosaic

by swyx 10mo ago

i intereviewed Jon (lead author on this paper) and yeah he pretty much disowns it now https://www.latent.space/p/mosaic-mpt-7b https://www.latent.space/p/mosaic-mpt-7b

gwern 10mo ago
Could you explain why you think that? I'm looking at the lottery ticket section and it seems like he doesn't disown it; the reason he gives, via Abhinav, for not pursuing it at his commercial job is just that that kind of sparsity is not hardware friendly (except with Cerebras). "It doesn't provide a speedup for normal commercial workloads on normal commercial GPUs and that's why I'm not following it up at my commercial job and don't want to talk about it" seems pretty far from "disowning the lottery ticket hypothesis [as wrong or false]".
- oofbey 10mo ago
  I think that was pretty clear even when this paper came out - even if you could find these sub networks they wouldn’t be faster on real hardware. Never thought much of this paper, but it sure did get a lot of people excited.
  - laughingcurve 10mo ago
    i saw how it nerdsniped an extremely capable faculty member
  - gwern 10mo ago
    (Cerebras is real hardware.)
    - deleted 10mo ago
      [deleted]
    - oofbey 10mo ago
      It is real in that it exists. It is not real in the sense that almost nobody has access to them. Unless you work at one of the handful of organizations with their hardware, it’s not a practical reality.
      - aaronblohowiak 10mo ago
        how long will that be the case?
        IshKebab 10mo ago
        At least for the foreseeable future (next 50 years say).
        oofbey 10mo ago
        They have a strange business model. Their chips are massive. So they necessarily only sell them to large customers. Also because of the way they’re built (entire wafer is a single chip) no two chips will be the same. Normally imperfections in the manufacturing result in some parts of the wafer being rejected and other binned as fast or slow chips. If you use the whole wafer you get what you get. So it’s necessarily a strange platform to work with - every device is slightly different.
  - sailingparrot 10mo ago
    It was exciting because of what it means regarding how a model learns, regardless on whether or not its commercially applicable.
- swyx 10mo ago
  he pretty much always says it offline haha but i maay have mixed it up with the subsequent convo we had at neurips https://www.latent.space/p/neurips-2023-startups https://www.latent.space/p/neurips-2023-startups
laughingcurve 10mo ago
cool beans, thanks for this -- I think it's easier to hear it directly from the authors. I was hesitant to start researchposting and come off like a dick. also; note to self: If I publish and disown my papers, shawn will interview me :)

by swyx 10mo ago

i intereviewed Jon (lead author on this paper) and yeah he pretty much disowns it now https://www.latent.space/p/mosaic-mpt-7b https://www.latent.space/p/mosaic-mpt-7b

gwern 10mo ago

Could you explain why you think that? I'm looking at the lottery ticket section and it seems like he doesn't disown it; the reason he gives, via Abhinav, for not pursuing it at his commercial job is just that that kind of sparsity is not hardware friendly (except with Cerebras). "It doesn't provide a speedup for normal commercial workloads on normal commercial GPUs and that's why I'm not following it up at my commercial job and don't want to talk about it" seems pretty far from "disowning the lottery ticket hypothesis [as wrong or false]".

oofbey 10mo ago
I think that was pretty clear even when this paper came out - even if you could find these sub networks they wouldn’t be faster on real hardware. Never thought much of this paper, but it sure did get a lot of people excited.
- laughingcurve 10mo ago
  i saw how it nerdsniped an extremely capable faculty member
- gwern 10mo ago
  (Cerebras is real hardware.)
  - deleted 10mo ago
    [deleted]
  - oofbey 10mo ago
    It is real in that it exists. It is not real in the sense that almost nobody has access to them. Unless you work at one of the handful of organizations with their hardware, it’s not a practical reality.
    - aaronblohowiak 10mo ago
      how long will that be the case?
      - IshKebab 10mo ago
        At least for the foreseeable future (next 50 years say).
      - oofbey 10mo ago
        They have a strange business model. Their chips are massive. So they necessarily only sell them to large customers. Also because of the way they’re built (entire wafer is a single chip) no two chips will be the same. Normally imperfections in the manufacturing result in some parts of the wafer being rejected and other binned as fast or slow chips. If you use the whole wafer you get what you get. So it’s necessarily a strange platform to work with - every device is slightly different.
- sailingparrot 10mo ago
  It was exciting because of what it means regarding how a model learns, regardless on whether or not its commercially applicable.
swyx 10mo ago
he pretty much always says it offline haha but i maay have mixed it up with the subsequent convo we had at neurips https://www.latent.space/p/neurips-2023-startups https://www.latent.space/p/neurips-2023-startups

laughingcurve 10mo ago

cool beans, thanks for this -- I think it's easier to hear it directly from the authors. I was hesitant to start researchposting and come off like a dick. also; note to self: If I publish and disown my papers, shawn will interview me :)