8 ms·

https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke https://huggingface.co/TheBloke There's a LocalL

https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke https://huggingface.co/TheBloke There's a LocalLLaMA subreddit, irc channels, and a whole big community around the web working on it on GitHub nd elsewhere. edit: I forgot to directly answer you: yes you can run these models. 16GB of plenty. Different quantizations give you different amounts of smarts and speed. There are tables that tell you how much RAM is needed per which quantization you choose, as well as how fast it can produce results (ms per token). e.g. https://github.com/ggerganov/llama.cpp#quantization https://github.com/ggerganov/llama.cpp#quantization where RAM required a little more than the file size, but there are tables that list it explicitly which I don't have immediately at hand.

tensor 3y ago
A reminder that llama isn't legal for the vast majority of use cases. Unless you signed their contract and then you can use it only for research purposes.
- rvcdbn 3y ago
  We don’t actually know that it’s not legal. The copyrightability of model weights is an open legal question right now afaik.
  - tensor 3y ago
    It doesn't have to be copyrightable to be intellectual property.
    - actionfromafar 3y ago
      Patents? Trademark? What do you mean?
      - Rexxar 3y ago
        Maybe this: https://en.wikipedia.org/wiki/Database_right https://en.wikipedia.org/wiki/Database_right but it doesn't exist in every countries.
    - twbarr 3y ago
      No, but what is it? Not your lawyer, not legal advice, but it's not a trade secret, they've given it to researchers. It's not a trademark because it's not an origin identifier. The structure might be patentable, but the weights won't be. It's certainly not a mask work. It might have been a contract violation for the guy who redistributed it, but I'm not a party to that contract.
      - Art9681 3y ago
        I'm going to play devil's advocate and state that a lot of what you mentioned will be relevant to a tiny part of the world that has the means to enforce this. The law will be forced to change as a response to AI. Many debates will be had. Many crap laws will be made by people grasping at straws but it's too late. Putting red tape around this technology puts that nation at a technological disadvantage. I would go as far as labeling a national security threat. I'm calling it now. Based on what I see today. Europe will position itself as a leader in AI legislation, and its economy will give way to the nations that want to enter the race and grab a chunk of the new economy. It's a Catch 22. You either gimp your own technological progress, or start a war with a nation that does not. Pretty sure Russia and China don't really care about the ethics behind it. There are plenty of nations capable enough in the same boat. Now what? OK, so in some hypothetical future China has an uncensored model with free reign over the internet. The US and Europe has banned this. What's stopping anyone from running the Chinese model? There isn't enough money in the world to enforce software laws. How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. If it did, OpenAI wouldn't exist.
        rileymat2 3y ago
        > How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. Forms of this argument get tossed out a lot. Laws don’t prevent, they hopefully limit. Murder has been illegal for a long time, it still happens.
    - rvcdbn 3y ago
      I’m going to go out on a limb here and assume that you’re making this statement because it feels like they should have some intellectual property rights in this case. Independently of whether that feeling corresponds to legal reality (the original question) I would also encourage you to question the source of this feeling. I believe it is rooted in an ideology where information is restricted as property by default. This is a dangerous ideology that constantly threatens to encroach on intellectual freedom e.g. software patents, gene patents. We have a wonderful tradition in the US that information is free by default. It has been much eroded by this ideology but I believe freedom is still legally the default unless the information falls under the criteria of trademark, copyright or patent. I think it’s important to recognize how this ideology of non-freedom has perniciously warped people’s default expectation around information sharing.
      - tensor 3y ago
        It has nothing to do with any sort of feeling. Perhaps you should check your own mental state. It is the same as any confidential data. Logs, readings from sensors, etc etc. If it's confidential and given to a 3rd party through a contract that doesn't mean that it's suddenly not confidential data for the rest of the world, even if the 3rd party leaks it. And if you really have a lawyer trying to tell you that some, at best, extreme grey area, is fine to build a business on, I think you should find a new lawyer.
        rvcdbn 3y ago
        I think that just further shows your worldview that defaults to information/data as property. I think this is wrong both in the sense that it isn't really what the law says (but aren't going to agree here anyway) but more importantly I think what it should say. Information should not be and is not property by default. There are only three specific ways in which it can become property ("intellectual property"): copyright, trademark and patent. If it's none of those then the government doesn't get to make any rules about how anyone deals in the data because of the 1st Amendment. That's my understanding of the US system at least.
- PostOnce 3y ago
  OpenLLaMA is though. https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama All of these are surmountable problems. We can beat OpenAI. We can drain their moat.
  - donw 3y ago
    For the above, are the RAM figures system RAM or GPU?
    - PostOnce 3y ago
      CPU RAM
  - kordlessagain 3y ago
    > We can drain their moat. I've got an AI powered sump pump if you need it.
    - ignoramous 3y ago
      They most certainly don't need / deserve the snark, to be sure, on hacker news of all places.
  - tensor 3y ago
    Absolutely, 100% agree. I just wouldn't touch the original LLaMA weights. There are many amazing open source models being built that should be used instead.
- niemandhier 3y ago
  It’s not clear if their license terms would hold, for the moment just act and worry later. Update: That is only true for the legal system I am currently residing in. No idea about e.g. the US.
- lhl 3y ago
  This is the most well-maintained list of commercially usable open LLMs: https://github.com/eugeneyan/open-llms https://github.com/eugeneyan/open-llms MPT, OpenLLaMA, and Falcon are probably the most generally useful. For code, Replit Code (specifically replit-code-instruct-glaive) and StarCoder (WizardCoder-15B) are the current top open models and both can be used commercially.

tensor 3y ago

A reminder that llama isn't legal for the vast majority of use cases. Unless you signed their contract and then you can use it only for research purposes.

rvcdbn 3y ago
We don’t actually know that it’s not legal. The copyrightability of model weights is an open legal question right now afaik.
- tensor 3y ago
  It doesn't have to be copyrightable to be intellectual property.
  - actionfromafar 3y ago
    Patents? Trademark? What do you mean?
    - Rexxar 3y ago
      Maybe this: https://en.wikipedia.org/wiki/Database_right https://en.wikipedia.org/wiki/Database_right but it doesn't exist in every countries.
  - twbarr 3y ago
    No, but what is it? Not your lawyer, not legal advice, but it's not a trade secret, they've given it to researchers. It's not a trademark because it's not an origin identifier. The structure might be patentable, but the weights won't be. It's certainly not a mask work. It might have been a contract violation for the guy who redistributed it, but I'm not a party to that contract.
    - Art9681 3y ago
      I'm going to play devil's advocate and state that a lot of what you mentioned will be relevant to a tiny part of the world that has the means to enforce this. The law will be forced to change as a response to AI. Many debates will be had. Many crap laws will be made by people grasping at straws but it's too late. Putting red tape around this technology puts that nation at a technological disadvantage. I would go as far as labeling a national security threat. I'm calling it now. Based on what I see today. Europe will position itself as a leader in AI legislation, and its economy will give way to the nations that want to enter the race and grab a chunk of the new economy. It's a Catch 22. You either gimp your own technological progress, or start a war with a nation that does not. Pretty sure Russia and China don't really care about the ethics behind it. There are plenty of nations capable enough in the same boat. Now what? OK, so in some hypothetical future China has an uncensored model with free reign over the internet. The US and Europe has banned this. What's stopping anyone from running the Chinese model? There isn't enough money in the world to enforce software laws. How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. If it did, OpenAI wouldn't exist.
      - rileymat2 3y ago
        > How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. Forms of this argument get tossed out a lot. Laws don’t prevent, they hopefully limit. Murder has been illegal for a long time, it still happens.
  - rvcdbn 3y ago
    I’m going to go out on a limb here and assume that you’re making this statement because it feels like they should have some intellectual property rights in this case. Independently of whether that feeling corresponds to legal reality (the original question) I would also encourage you to question the source of this feeling. I believe it is rooted in an ideology where information is restricted as property by default. This is a dangerous ideology that constantly threatens to encroach on intellectual freedom e.g. software patents, gene patents. We have a wonderful tradition in the US that information is free by default. It has been much eroded by this ideology but I believe freedom is still legally the default unless the information falls under the criteria of trademark, copyright or patent. I think it’s important to recognize how this ideology of non-freedom has perniciously warped people’s default expectation around information sharing.
    - tensor 3y ago
      It has nothing to do with any sort of feeling. Perhaps you should check your own mental state. It is the same as any confidential data. Logs, readings from sensors, etc etc. If it's confidential and given to a 3rd party through a contract that doesn't mean that it's suddenly not confidential data for the rest of the world, even if the 3rd party leaks it. And if you really have a lawyer trying to tell you that some, at best, extreme grey area, is fine to build a business on, I think you should find a new lawyer.
      - rvcdbn 3y ago
        I think that just further shows your worldview that defaults to information/data as property. I think this is wrong both in the sense that it isn't really what the law says (but aren't going to agree here anyway) but more importantly I think what it should say. Information should not be and is not property by default. There are only three specific ways in which it can become property ("intellectual property"): copyright, trademark and patent. If it's none of those then the government doesn't get to make any rules about how anyone deals in the data because of the 1st Amendment. That's my understanding of the US system at least.
PostOnce 3y ago
OpenLLaMA is though. https://github.com/openlm-research/open_llama https://github.com/openlm-research/open_llama All of these are surmountable problems. We can beat OpenAI. We can drain their moat.
- donw 3y ago
  For the above, are the RAM figures system RAM or GPU?
  - PostOnce 3y ago
    CPU RAM
- kordlessagain 3y ago
  > We can drain their moat. I've got an AI powered sump pump if you need it.
  - ignoramous 3y ago
    They most certainly don't need / deserve the snark, to be sure, on hacker news of all places.
- tensor 3y ago
  Absolutely, 100% agree. I just wouldn't touch the original LLaMA weights. There are many amazing open source models being built that should be used instead.
niemandhier 3y ago
It’s not clear if their license terms would hold, for the moment just act and worry later. Update: That is only true for the legal system I am currently residing in. No idea about e.g. the US.
lhl 3y ago
This is the most well-maintained list of commercially usable open LLMs: https://github.com/eugeneyan/open-llms https://github.com/eugeneyan/open-llms MPT, OpenLLaMA, and Falcon are probably the most generally useful. For code, Replit Code (specifically replit-code-instruct-glaive) and StarCoder (WizardCoder-15B) are the current top open models and both can be used commercially.