7 ms·

> The training code, dataset and weights for this model are open sourced so that developers can reproduce the model and help train other SLMs and LLMs. Wow, an

by diggan 2y ago

> The training code, dataset and weights for this model are open sourced so that developers can reproduce the model and help train other SLMs and LLMs. Wow, an actual open source language model (first of its kind [from a larger company] maybe even?), includes all you need to be able to recreate it from scratch. Thanks AMD! Available under this funky GitHub organization it seems: https://github.com/AMD-AIG-AIMA/AMD-LLM https://github.com/AMD-AIG-AIMA/AMD-LLM

bubaumba 2y ago
No, it's not open source till someone can actually reproduce it. That's the hardest part. For now it's open weights open dataset. Which is not the same.
- diggan 2y ago
  That's... Not how open source works? The "binary" (model weights) is open source and the "software" (training scripts + data used for training) is open source, this release is a real open source release. Independent reproduction is not needed to call something open source. Can't believe it's the second time I end up with the very same argument about what open source is today on HN.
  - dboreham 2y ago
    But wouldn't failure to achieve independent reproduction falsify the open claim? Similar to you publish the source for Oracle (the database), but nobody can build a binary from it because it needs magic compliers or test suites that aren't open source? Heck when the browser was open-sourced, there was an explicit test where the source was given to some dude who didn't work for Netscape to verify that he could actually make a working binary. It's a scene in the movie "Code Rush".
  - bubaumba 2y ago
    You are missing key points here. "reproduce" means produce the same. Not just train similar model. I can simplify the task, can you convincingly explain how the same model can be produced from this dataset? We can start simple, how you can possibly get the same weights after the first single iteration? I.e. the same as original model got. Pay attention to randomness, data selection, initial model state. Ok, if you can't do that. Can you explain in believable way how to prove that given model was trained on give dataset? I'm not asking you for actually doing all these things, that could be expensive, only to explain how it can be done. Strict 'open source' includes not only open weights, open data. It also includes the word "reproducible". It's not "reproduced", only "reproducible". And even this is not the case here.
    - worewood 2y ago
      How often do people expect to compile open-source code and get _exactly_ the same binary as the distributed one? I've seen this kind of restriction only on decompilation projects e.g. the SM64 decompilation -- where they deliberately compare the hashes of original vs. compiled binaries, as a way to verify the decompilation is correct. It's an unreasonable request with ordinary code, even more with ML where very few ones have access to the necessary hardware, and where in practice, it is not deterministic.
      - e12e 2y ago
        I expect that if I compile your 3d renderer, and feed it the same scene file you did - I get the same image?
        TylerE 2y ago
        Why would you expect that? 3D renderers are not generally deterministic. Many will incorporate, for instance, noise algorithms. They will frequently not produce byte-identical renders on the same hardware using the same binary.
        e12e 2y ago
        Same recognizable image? Like if you look at the povray benchmark image on Linux and Windows? For an llm that should (ideally) translate to similar answers?
      - bubaumba 2y ago
        > How often do people expect to compile open-source code and get _exactly_ the same binary _Always_, with the right options. And that's the key point. If distributed code is different it means it may be infected or altered in other way. In other words it cannot be trusted. The same with models, if they are not reproducible or verifiable they cannot be trusted. Trust is the main feature of open source. Calling black box with attached data 'open source', even 'the first' is a bit of a stretch. It's not reproducible and not verifiable. And it's definitely not the first model with open data. To be correct you should add 'untrusted' if you want to call this thing 'open source'. Like with Meta's models who knows what it holds. PS: finally I'm negative, fanboys don't like it ;-)
  - wrs 2y ago
    The interesting part of the product we’re taking about (that is, the equivalent of the executable binary of an ordinary software product) is the weights. The “source” is not sufficient to “recompile” the product (i.e., recreate the weights). Therefore, while the source you got is open, you didn’t get all the source to the thing that was supposedly “open source”. It’s like if I said I open-sourced the Matrix trilogy and only gave you the DVD image and the source to the DVD decoder. (Edit: Sorry, I replied to the wrong comment. I’m talking primarily about the typical sort of release we see, not this one which is a lot closer to actually open.)
    - littlestymaar 2y ago
      > The “source” is not sufficient to “recompile” the product (i.e., recreate the weights). Therefore, while the source you got is open, you didn’t get all the source to the thing that was supposedly “open source”. What's missing?
      - wrs 2y ago
        Well, I’m not experienced in training full-sized LLMs, and it’s conceivable that in this particular case the training process is simple enough that nothing is missing. That would be a rarity, though. But see my edit above — I’m not actually reacting to this release when I say that.
        littlestymaar 2y ago
        OK, so you just like to be a contrarian…
- Jabrov 2y ago
  What’s the difference?
  - avaldez_ 2y ago
    Reproducibility? I mean what's the point of an open technology nobody knows if it works or not.
  - frontalier 2y ago
    the goal posts moved?!
jerrygenser 2y ago
This would be another example of open source. Not from such a large company but a good reference including code, data, weights, etc. https://allenai.org/olmo https://allenai.org/olmo
- brianjking 2y ago
  Molmo even more so! The 7b is wild.
wrs 2y ago
We (developers and tech managers) really need to hold the line on this terminology. This is a full actual open source LLM. The usual “open inference” model is not.
- boulos 2y ago
  I assume by "open inference" you mostly mean "weights available"?
  - wrs 2y ago
    Usually “open source” for an LLM means you get the weights and the inference code, which I’ve started calling “open inference”. It’s certainly good and useful, but it’s not the actual source of the model. I find people get into silly arguments about the terminology because they’re focused on whether the “source” is “open” and not on what the “source” is actually the source of. “Weights available” indicates even the weights aren’t “open” in the usual software meaning of the term, as they typically come with restrictive licenses (more restrictive than copyleft or attribution).
    - throwawaymaths 2y ago
      By "source" do you mean training data?
  - nickpsecurity 2y ago
    I call them open weights or just freeware like when we got only the EXE’s on Windows.
- wmf 2y ago
  You're not wrong, but if you come up with a definition that no one is willing to meet you're just making that definition irrelevant.
  - deleted 2y ago
    [deleted]
GeekyBear 2y ago
> Wow, an actual open source language model (first of its kind Apple research has previously released another example of a model with open training code, data, and weights, but their model was sized for running inference workloads on mobile devices. However, Apple has a mobile device line of business and AMD has an enterprise AI accelerator line of business, so they are both doing work relevant to their bottom line.
- diggan 2y ago
  Thanks, seems you're talking about the OpenELM family of models: https://github.com/apple/corenet/tree/main/projects/openelm https://github.com/apple/corenet/tree/main/projects/openelm
kypro 2y ago
Smart move from AMD. Helps develop an ecosystem around their tech and for their GPUs.
- jeff_carr 2y ago
  Has anyone tried it? I mean, I would, but as far as I can tell understand I need 4 boxes with 4 GPU's. Plus an interconnect. I mean, I could put in an order for my homelab but at around $80k per box and maybe $20k for the right switches and some other gear, my wife will probably frown at me ordering a $340,000 rig to try this code that I don't know what to do with it if it works. Maybe some other heavy hitter out there can explain what all this whatchamacallit newfangled synergy producing matrix algebra does after you have it running?
  - Shadowmist 2y ago
    > that I don't know what to do with it if it works. After you get it up and running you can just ask it what to do with it.
    - Rinzler89 2y ago
      Hope it tells you to buy more AMD HW because that would be so funny.
  - kypro 2y ago
    Seems kinda obvious this isn't targeted at hobbyists, but more towards SMEs and AI startups that want to build their own language models from scratch or experiment with the tech. Something like this would help small teams build an initial POC and do some experimentation. You have similar issues with robotics projects. It's very expensive for a hobbyist because of the hardware costs, but there's large number of small companies who benefit from open source tech to get their projects started.
NitpickLawyer 2y ago
> Wow, an actual open source language model I find it funny that the AI field has somehow normalised the goalpost moving from capabilities all the way to definitions about open source. And people seem really tribal about it... There absolutely are open source LLMs already. Phi3.5 (MIT), various Mistral models (Apache2.0), various Qwen2 models (Apache2.0) and so on. LLamas are not open source, nor are Gemmas. But to say this is "an actual open source model" is weird nitpicking for the sake of nitpicking, IMO. Requiring the methods and datasets that someone used to create some piece of IP is in no way a requirement for open sourcing said IP. It never has been! Imagine this analogy: A dev comes up with a way to generate source code that solves a real problem. This dev uses a secret seed, that only they know. The dev also uses thousands of hours of compute, and an algorithm that they created. At the end of the exercise they release the results on github, as follows: - here is a project that takes in a piece of text in english, and translates it into french. - the resulting source code is massive. 10 billions LOC. The lines of code are just if statements, all the way down, with some hardcoded integer values. - source code licensed under Apache 2.0, written in let's say python. - users can see the source code - users can run the source code - users can modify the source code and re-release the code Now, would anyone pre LLMs say "this isn't true open source" because it's too complicated? Because no one can reasonably understand the source code? Because it uses hard coded int values? Because it's 10b LOC? Because the dev never shared how they got those values? Of course not. The resulting code would have been open source because Apache 2.0 is open source. It's the same with model weights. Just because they're not source code, and just because you don't know how they were created, it does not mean the weights are not open source. You can see the weights. You can change the weights. You can re-distribute the weights. It's open source. The definition of something being open source does not cover you understanding why the weights are like they are. Nor do they require you having access to the methods of creating those weights. Or datasets. Or whatever the devs had for breakfast.
- diggan 2y ago
  > that the AI field has somehow normalised the goalpost moving from capabilities all the way to definitions about open source The problem is that Facebook and others are trying to move the goalpost, while others like me would like the goalpost to remain where it is, namely we call projects "Open source" when the required parts to build it on our own machines, is sufficiently accessible. As I probably wouldn't be a developer in the first place if it wasn't for FOSS, and I spend literally all day long contributing to others FOSS projects and working on my own, it's kind of scary seeing these large companies trying to change what FOSS means. I think you're forgetting about the intent and purpose of open source. The goal is that people can run software for whatever purpose they want, and they can modify it for whatever purpose. This is the intent behind the licenses we use when we "create FOSS". This means, in practice, that the source code has to be accessible somehow, so the compiler I have on my computer, can build a similar binary to the one the project itself offers (if it does). The source code has to be accessible so I can build the project, but also modify it for myself. Taking this idea that mostly only applied to software before (FOSS) but applying it to ML instead, it's clear to see what we need in order to 1) be able to use it as we want and 2) be able to modify it as we want. > You can see the weights. You can change the weights. You can re-distribute the weights. It's open source. Right. If I upload a binary to some website, you can see the binary, you can change the binary and you can re-distribute it. Would you say the binary is open source? The weights are the binary in ML contexts. It's OK for projects to publish those weights, but it's not OK to suddenly change the definition and meaning of open source because companies want to look like they're doing FOSS, when in reality they're publishing binaries without any ways of building those binaries with your own changes. Imagine if the Linux kernel was just a big binary blob. Yes, you can change it, re-distribute and what not, but only in a binary-blob shape. You'd be kind of out there if you insist on calling this binary-blob kernel FOSS. I'm sure you'd be able to convince some Facebook engineers about it, seems they're rolling with that idea already, but the rest of us who exist in the FOSS ecosystem? We'd still have the same goalpost in the exact same spot it's been for at least two decades I've been involved.

by diggan 2y ago

bubaumba 2y ago

No, it's not open source till someone can actually reproduce it. That's the hardest part. For now it's open weights open dataset. Which is not the same.

diggan 2y ago
That's... Not how open source works? The "binary" (model weights) is open source and the "software" (training scripts + data used for training) is open source, this release is a real open source release. Independent reproduction is not needed to call something open source. Can't believe it's the second time I end up with the very same argument about what open source is today on HN.
- dboreham 2y ago
  But wouldn't failure to achieve independent reproduction falsify the open claim? Similar to you publish the source for Oracle (the database), but nobody can build a binary from it because it needs magic compliers or test suites that aren't open source? Heck when the browser was open-sourced, there was an explicit test where the source was given to some dude who didn't work for Netscape to verify that he could actually make a working binary. It's a scene in the movie "Code Rush".
- bubaumba 2y ago
  You are missing key points here. "reproduce" means produce the same. Not just train similar model. I can simplify the task, can you convincingly explain how the same model can be produced from this dataset? We can start simple, how you can possibly get the same weights after the first single iteration? I.e. the same as original model got. Pay attention to randomness, data selection, initial model state. Ok, if you can't do that. Can you explain in believable way how to prove that given model was trained on give dataset? I'm not asking you for actually doing all these things, that could be expensive, only to explain how it can be done. Strict 'open source' includes not only open weights, open data. It also includes the word "reproducible". It's not "reproduced", only "reproducible". And even this is not the case here.
  - worewood 2y ago
    How often do people expect to compile open-source code and get _exactly_ the same binary as the distributed one? I've seen this kind of restriction only on decompilation projects e.g. the SM64 decompilation -- where they deliberately compare the hashes of original vs. compiled binaries, as a way to verify the decompilation is correct. It's an unreasonable request with ordinary code, even more with ML where very few ones have access to the necessary hardware, and where in practice, it is not deterministic.
    - e12e 2y ago
      I expect that if I compile your 3d renderer, and feed it the same scene file you did - I get the same image?
      - TylerE 2y ago
        Why would you expect that? 3D renderers are not generally deterministic. Many will incorporate, for instance, noise algorithms. They will frequently not produce byte-identical renders on the same hardware using the same binary.
        e12e 2y ago
        Same recognizable image? Like if you look at the povray benchmark image on Linux and Windows? For an llm that should (ideally) translate to similar answers?
    - bubaumba 2y ago
      > How often do people expect to compile open-source code and get _exactly_ the same binary _Always_, with the right options. And that's the key point. If distributed code is different it means it may be infected or altered in other way. In other words it cannot be trusted. The same with models, if they are not reproducible or verifiable they cannot be trusted. Trust is the main feature of open source. Calling black box with attached data 'open source', even 'the first' is a bit of a stretch. It's not reproducible and not verifiable. And it's definitely not the first model with open data. To be correct you should add 'untrusted' if you want to call this thing 'open source'. Like with Meta's models who knows what it holds. PS: finally I'm negative, fanboys don't like it ;-)
- wrs 2y ago
  The interesting part of the product we’re taking about (that is, the equivalent of the executable binary of an ordinary software product) is the weights. The “source” is not sufficient to “recompile” the product (i.e., recreate the weights). Therefore, while the source you got is open, you didn’t get all the source to the thing that was supposedly “open source”. It’s like if I said I open-sourced the Matrix trilogy and only gave you the DVD image and the source to the DVD decoder. (Edit: Sorry, I replied to the wrong comment. I’m talking primarily about the typical sort of release we see, not this one which is a lot closer to actually open.)
  - littlestymaar 2y ago
    > The “source” is not sufficient to “recompile” the product (i.e., recreate the weights). Therefore, while the source you got is open, you didn’t get all the source to the thing that was supposedly “open source”. What's missing?
    - wrs 2y ago
      Well, I’m not experienced in training full-sized LLMs, and it’s conceivable that in this particular case the training process is simple enough that nothing is missing. That would be a rarity, though. But see my edit above — I’m not actually reacting to this release when I say that.
      - littlestymaar 2y ago
        OK, so you just like to be a contrarian…
Jabrov 2y ago
What’s the difference?
- avaldez_ 2y ago
  Reproducibility? I mean what's the point of an open technology nobody knows if it works or not.
- frontalier 2y ago
  the goal posts moved?!

jerrygenser 2y ago

This would be another example of open source. Not from such a large company but a good reference including code, data, weights, etc. https://allenai.org/olmo https://allenai.org/olmo

brianjking 2y ago
Molmo even more so! The 7b is wild.

wrs 2y ago

We (developers and tech managers) really need to hold the line on this terminology. This is a full actual open source LLM. The usual “open inference” model is not.

boulos 2y ago
I assume by "open inference" you mostly mean "weights available"?
- wrs 2y ago
  Usually “open source” for an LLM means you get the weights and the inference code, which I’ve started calling “open inference”. It’s certainly good and useful, but it’s not the actual source of the model. I find people get into silly arguments about the terminology because they’re focused on whether the “source” is “open” and not on what the “source” is actually the source of. “Weights available” indicates even the weights aren’t “open” in the usual software meaning of the term, as they typically come with restrictive licenses (more restrictive than copyleft or attribution).
  - throwawaymaths 2y ago
    By "source" do you mean training data?
- nickpsecurity 2y ago
  I call them open weights or just freeware like when we got only the EXE’s on Windows.
wmf 2y ago
You're not wrong, but if you come up with a definition that no one is willing to meet you're just making that definition irrelevant.
- deleted 2y ago
  [deleted]

GeekyBear 2y ago

> Wow, an actual open source language model (first of its kind Apple research has previously released another example of a model with open training code, data, and weights, but their model was sized for running inference workloads on mobile devices. However, Apple has a mobile device line of business and AMD has an enterprise AI accelerator line of business, so they are both doing work relevant to their bottom line.

diggan 2y ago
Thanks, seems you're talking about the OpenELM family of models: https://github.com/apple/corenet/tree/main/projects/openelm https://github.com/apple/corenet/tree/main/projects/openelm

kypro 2y ago

Smart move from AMD. Helps develop an ecosystem around their tech and for their GPUs.

jeff_carr 2y ago
Has anyone tried it? I mean, I would, but as far as I can tell understand I need 4 boxes with 4 GPU's. Plus an interconnect. I mean, I could put in an order for my homelab but at around $80k per box and maybe $20k for the right switches and some other gear, my wife will probably frown at me ordering a $340,000 rig to try this code that I don't know what to do with it if it works. Maybe some other heavy hitter out there can explain what all this whatchamacallit newfangled synergy producing matrix algebra does after you have it running?
- Shadowmist 2y ago
  > that I don't know what to do with it if it works. After you get it up and running you can just ask it what to do with it.
  - Rinzler89 2y ago
    Hope it tells you to buy more AMD HW because that would be so funny.
- kypro 2y ago
  Seems kinda obvious this isn't targeted at hobbyists, but more towards SMEs and AI startups that want to build their own language models from scratch or experiment with the tech. Something like this would help small teams build an initial POC and do some experimentation. You have similar issues with robotics projects. It's very expensive for a hobbyist because of the hardware costs, but there's large number of small companies who benefit from open source tech to get their projects started.

NitpickLawyer 2y ago

> Wow, an actual open source language model I find it funny that the AI field has somehow normalised the goalpost moving from capabilities all the way to definitions about open source. And people seem really tribal about it... There absolutely are open source LLMs already. Phi3.5 (MIT), various Mistral models (Apache2.0), various Qwen2 models (Apache2.0) and so on. LLamas are not open source, nor are Gemmas. But to say this is "an actual open source model" is weird nitpicking for the sake of nitpicking, IMO. Requiring the methods and datasets that someone used to create some piece of IP is in no way a requirement for open sourcing said IP. It never has been! Imagine this analogy: A dev comes up with a way to generate source code that solves a real problem. This dev uses a secret seed, that only they know. The dev also uses thousands of hours of compute, and an algorithm that they created. At the end of the exercise they release the results on github, as follows: - here is a project that takes in a piece of text in english, and translates it into french. - the resulting source code is massive. 10 billions LOC. The lines of code are just if statements, all the way down, with some hardcoded integer values. - source code licensed under Apache 2.0, written in let's say python. - users can see the source code - users can run the source code - users can modify the source code and re-release the code Now, would anyone pre LLMs say "this isn't true open source" because it's too complicated? Because no one can reasonably understand the source code? Because it uses hard coded int values? Because it's 10b LOC? Because the dev never shared how they got those values? Of course not. The resulting code would have been open source because Apache 2.0 is open source. It's the same with model weights. Just because they're not source code, and just because you don't know how they were created, it does not mean the weights are not open source. You can see the weights. You can change the weights. You can re-distribute the weights. It's open source. The definition of something being open source does not cover you understanding why the weights are like they are. Nor do they require you having access to the methods of creating those weights. Or datasets. Or whatever the devs had for breakfast.

diggan 2y ago
> that the AI field has somehow normalised the goalpost moving from capabilities all the way to definitions about open source The problem is that Facebook and others are trying to move the goalpost, while others like me would like the goalpost to remain where it is, namely we call projects "Open source" when the required parts to build it on our own machines, is sufficiently accessible. As I probably wouldn't be a developer in the first place if it wasn't for FOSS, and I spend literally all day long contributing to others FOSS projects and working on my own, it's kind of scary seeing these large companies trying to change what FOSS means. I think you're forgetting about the intent and purpose of open source. The goal is that people can run software for whatever purpose they want, and they can modify it for whatever purpose. This is the intent behind the licenses we use when we "create FOSS". This means, in practice, that the source code has to be accessible somehow, so the compiler I have on my computer, can build a similar binary to the one the project itself offers (if it does). The source code has to be accessible so I can build the project, but also modify it for myself. Taking this idea that mostly only applied to software before (FOSS) but applying it to ML instead, it's clear to see what we need in order to 1) be able to use it as we want and 2) be able to modify it as we want. > You can see the weights. You can change the weights. You can re-distribute the weights. It's open source. Right. If I upload a binary to some website, you can see the binary, you can change the binary and you can re-distribute it. Would you say the binary is open source? The weights are the binary in ML contexts. It's OK for projects to publish those weights, but it's not OK to suddenly change the definition and meaning of open source because companies want to look like they're doing FOSS, when in reality they're publishing binaries without any ways of building those binaries with your own changes. Imagine if the Linux kernel was just a big binary blob. Yes, you can change it, re-distribute and what not, but only in a binary-blob shape. You'd be kind of out there if you insist on calling this binary-blob kernel FOSS. I'm sure you'd be able to convince some Facebook engineers about it, seems they're rolling with that idea already, but the rest of us who exist in the FOSS ecosystem? We'd still have the same goalpost in the exact same spot it's been for at least two decades I've been involved.