16 ms·

Hey HN, we've released tools that make it easy to test LLaMa 2 and add it to your own app! Model playground here: https://llama2.ai https://llama2.ai Hosted c

• refulgentis 3y ago
  Seeing a16z w/early access, enough to build multiple tools in advance, is a very unpleasant reminder of insularity and self-dealing of SV elites. My greatest hope for AI is no one falls for this kind of stuff the way we did for mobile.
  • dicishxg 3y ago
    And yet here we are a few weeks after that with a free to use model that cost millions to develop and is open to everyone. I think you’re taking an unwarranted entitled view.
    • refulgentis 3y ago
      I can't parse this: I assume it assumes I assume that a16z could have ensured it wasn't released It's not that, just what it says on the tin: SV elites are not good for SV
    • ipaddr 3y ago
      You act like this is a gift of charity instead of attempts to stay relevant.
      • ianbutler 3y ago
        What? Tell me you don't follow the space. FB AI is one of the top labs..
        • refulgentis 3y ago
          We're talking about a16z, not Facebook.
      • unshavedyak 3y ago
        The best charity is one that benefits both parties, no?
        • TechBro8615 3y ago
          That's just a trade. If we assume "charity" is "altruism," then by definition there must be no benefit to the giver.
          • freedomben 3y ago
            I don't think that's even possible, but if it was it would be a disaster because humans don't work that way. We respond to incentive. When giving to charity, the incentive can be as simple as "I feel good" but it's still an incentive.
            • ipaddr 3y ago
              Some do what's right even if it doesn't feel good. The best charity can be painful.
              • freedomben 3y ago
  • whimsicalism 3y ago
    e: Oh - this is a16z, so yeah probably early access - scratch my additional comments I agree that I don't like early/insider stuff That said - I believe Llama 2 is architecturally identical to the previous one and given that they are using 13B it is probably just a drag and drop bin replacement and reload your servers. We all knew Llama 2 was coming so it might be within the capabilities of a hungry startup with no early access.
  • sfmike 3y ago
    Any entity working on something in beta has early access anyone could hypothetically email out in grunt work and get involved in a project and get early access therefore this logic when tested against a third party example such as any engineer this argument isn't valid as any engineer entitled or not could have early access
    • ipaddr 3y ago
      Anyone could worm their way into the project so it's not closed? It's a closed beta with invites to a select few. Some could cross that barrier but the truth is very few will be invited to the inner circle. This started as a private project and painted itself with an open source brush for advertising.
  • blululu 3y ago
    Asking someone to beta test your product is more like asking them for a favor than the other way around. Finding a sympathetic person to try out your stuff and provide intelligent, relevant feedback in a timely fashion is not that easy.
• swyx 3y ago
  its not clear but can we also download the model with this Llama v2 Cog thing? EDIT: Meta is being extremely prompt, just got sent the download instructions https://twitter.com/swyx/status/1681351712718876673 https://twitter.com/swyx/status/1681351712718876673 also is it now Llama or LLaMA since the website says Llama? lol
  • sanjarbek16 3y ago
    Meta sent me the link for the weights right away after I filled out the form.
  • rvz 3y ago
    >its not clear but can we also download the model with this Llama v2 Cog thing? why did replicate/a16z seem to have the model but the rest of us have to wait to download the weights? It's a club and we're not invited. Just like what OpenAI did with early access with so-called AI startups with YC, Meta has done the same with the VC grifters. Money is power and buys you into their club.
  • tyre 3y ago
    > the rest of us have to wait to download the weights they're openly torrentable
  • imjonse 3y ago
    Probaly Llama, it was too hard to get right before :) https://github.com/facebookresearch/llama/commit/6d4c0c290aeec1fa4399694fefb864be5a153bb6 https://github.com/facebookresearch/llama/commit/6d4c0c290ae...
  • capableweb 3y ago
    Seems people here on HN were lucky. Signed up ~20 minutes ago and no download link, so surely they're doing some checking, it's not fully automated, unlike what others say here. On Huggingface, the following disclaimer has been put in place: > This is a form to enable access to Llama 2 on Hugging Face after you have been granted access from Meta. Please visit the [Meta website](https://ai.meta.com/resources/models-and-libraries/llama-downloads https://ai.meta.com/resources/models-and-libraries/llama-dow...) and accept our license terms and acceptable use policy before submitting this form. Requests will be processed in 1-2 days https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/tree/main https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/tree/m... So seems it's a manual flow on their side to "accept" downloads.
• LoganDark 3y ago
  > Model playground here: llama2.ai is there any way to rename the model from "Assistant" to tune out those "I am a large language model" excuses?
  • delfinom 3y ago
    Lol, so I kept spamming "no" and it eventually leaked it's boot instruction which was "Please respond as Assistant" So, just ask the LLM to respond as something else ;)
    • rajko_rad 3y ago
      We were trying to do the same thing once deployed haha... we found, if you want it to take on another persona, you could end your prompt as: 'Assistant: (responding as XYZXYZ)' And then let it complete! Worked much more reliably than just saying: 'XYZXYZ:'
  • codethief 3y ago
    In the menu there's an option to modify the prompt.
    • LoganDark 3y ago
      > In the menu there's an option to modify the prompt. Yes, that's the first thing I saw, but there's no way to tell the chat formatter to put something other than "Assistant:" at the end for completions.
• deleted 3y ago
  [deleted]
• Aardwolf 3y ago
  Will Llama 2 also work as a drop-in in existing tools like llama.cpp, or does it require different / updated tools?
  • woadwarrior01 3y ago
    Not quite a drop in replacement, but close enough. From the paper[1]: > Llama 2, an updated version of Llama 1, trained on a new mix of publicly available data. We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023)[2]. [1]: https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/ https://ai.meta.com/research/publications/llama-2-open-found... [2]: https://arxiv.org/abs/2305.13245 https://arxiv.org/abs/2305.13245
  • swyx 3y ago
    some adaptation needed https://github.com/ggerganov/llama.cpp/issues/2262 https://github.com/ggerganov/llama.cpp/issues/2262
• avaer 3y ago
  How does one apply for a job with the the internal A16Z teams experimenting with this?
  • meroes 3y ago
    Ask Llama of course. Showing that you are willing to ask an LLM is a perfect sign for a candidate!
    • heyoni 3y ago
      It’d be fun if they added Easter eggs to it just like how companies would advertise jobs in the browser console.
• arbuge 3y ago
  I like the way the playground allows easy modification of the system prompt. I suggest adding "You are very cranky." to the default prompt for interesting results.
  • nottheengineer 3y ago
    Holy shit, I've never seen an AI go schizophrenic this hard. That's my first try: https://gist.github.com/miguel7501/983f794e13cc762eb6274c9b23ebca5e https://gist.github.com/miguel7501/983f794e13cc762eb6274c9b2...
    • d0mine 3y ago
      Every digit is equally likely in pi: >>> statistics.mean(map(int, string.digits)) 4.5
• baby 3y ago
  I'm wondering how do people compare different models? I've been trying chatGPT 3.5, bing chat (chatgpt 4 I believe?), and bard, and now this one, and I'm not sure if there's a noticeable difference in terms of "this is better"
  • dotancohen 3y ago
    Depends on the task. For code, ask it to implement a not-difficult but not-trivial feature. "Please add hooks to the AnkiDroid source code so that addons would be viable" might be a good start, for something that is on my mind. Then compare implementations. For checking hallucinations, ask it about events and trivia that happened eons ago, and also within the last decade. Try some things that it can not possibly know, like how much celery Brad Pitt likes in his salad.
    • rajko_rad 3y ago
      This is an emerging space with lots of interesting tools coming out... There are many established benchmarks out there (i.e. included on front page of llama2 release), but most product builders have their own sets of evals that are more relevant to them... Here is a thread exploring differences between llama-v2 vs. gpt3.5: https://twitter.com/rajko_rad/status/1681344850510376960 https://twitter.com/rajko_rad/status/1681344850510376960
  • losteric 3y ago
    Develop a set of queries for the use-case with human review of outputs. My team has an internal (corporate) tool where we drop in an S3 file, complete text over K models, then evaluate the completions with appropriate humans labor pools. Each evaluator gets a pair of outputs for the same prompt and picks the best.
  • jimmySixDOF 3y ago
    Try the Chat Arena with ELO ratings based on end user side by side blind tests. It run out of UC Berkley by lmsys the same team that released Vicuna. https://arena.lmsys.org/ https://arena.lmsys.org/
• tmaly 3y ago
  How are the model weights licensed?
• Oras 3y ago
  Is it expected to be slow? Each request takes several seconds to reply. Also how recent is the training data? When I asked what’s the date today? I received 22nd of March 2018. Interesting choice of using streamlit for interface.
  • punk_ihaq 3y ago
    > Is it expected to be slow? Probably, yes. The slowness is not on the Streamlit end, but on the Replicate API end. The docs for the 13b API [0] say: > Predictions typically complete within 9 seconds. Whereas for the 70b API [1]: > Predictions typically complete within 18 seconds. The predict time for this model varies significantly based on the inputs. [0] https://replicate.com/a16z-infra/llama13b-v2-chat https://replicate.com/a16z-infra/llama13b-v2-chat [1] https://replicate.com/replicate/llama70b-v2-chat https://replicate.com/replicate/llama70b-v2-chat
• herpderperator 3y ago
  ...is this supposed to make sense? https://www.dropbox.com/scl/fi/5j0mtwdrqm08dxbh84hh6/Screen-Shot-2023-07-18-at-6.58.56-PM.png?rlkey=7s43dqqvny4n6vjkcx7alzoty&dl=0 https://www.dropbox.com/scl/fi/5j0mtwdrqm08dxbh84hh6/Screen-...
  • derefr 3y ago
    You're expecting a language model (as opposed to a more general form of ML model) to be numerate? It has no idea that the token "$105,000" has a relationship "is less than" with the token "$140,000". It probably knows that the token "1" has the relationship "is less than" with the token "2" — but that's because it has "1" and "2" as reified concepts, each with many different facts and properties and relationships associated directly with those tokens-as-vertices. "$105,000", meanwhile, is just a lexeme. It maybe knows, due to pre-parsing, that it's "an amount of dollars" — and maybe it even recognizes its order-of-magnitude. It can therefore likely make any statement that takes the token "$105,000" as a meta-syntactic variable standing in for some unknown "amount of dollars." But there's no little numeric model embedded inside the language model that would tell it how many dollars, or be able to compare dollars against dollars.
    • sebzim4500 3y ago
      The problem with this kind of simplistic analysis is that if you ask llama which of those numbers is larger it will of course get it right. Clearly there is some understanding of the relative size of numbers, since it can compare numbers that could not possibly have been in the training set.
      • codethief 3y ago
        I think you're both right and, as always, the truth is to be found somewhere in the middle: > Which number is greater, 105,000 or 140,000? > 140,000 is greater than 105,000. > How do you know? > 140,000 is greater than 105,000 because 140,000 is 35,000 more than 105,000. But compare this to: > Which number is greater: 12345 or 112345? > 112345 is greater than 12345. > How do you know? > 112345 has more digits than 12345. > What about 123456 and 134567? > 134567 is greater than 123456. > How do you know? > 134567 has more digits than 123456. > But they have the same number of digits: Six. > 134567 has more significant digits than 123456. > What do you mean by that? > 134567 has more leading zeros than 123456. > But neither has any zero. > 134567 has more digits in total than 123456. > But that's not true. They have the same number of digits. Yet, it is true that 134567 is greater than 123456, as you say. But why? > 134567 is greater than 123456 because it has more digits in total. Another attempt, with smaller context size: > Which number is greater, 123456 or 134567? > 134567 is greater than 123456. > How do you know? > 134567 is greater because 123456 has 6 digits while 134567 has 7 digits. > But 134567 has 6 digits > 134567 has 7 digits.
• villgax 3y ago
  You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof). WTF kinda open for commercial model licensing is this
  • nottheengineer 3y ago
    The "GPL but just for me" apparently. Not sure how they're going to monetize the monopoly they are trying to secure.
• arkano 3y ago
  I appreciate that the playground frontend is just a streamlit app.
• jug 3y ago
  Still fails my hippo test! > Yes, hippos are excellent swimmers. They spend most of their time in the water, where they feed on aquatic plants and escape the heat of the savannah. In fact, hippos are one of the best swimmers among all land mammals. But that's fine. Most do. Hippos don't swim. They walk or hop/skip at best underwater.
  • Jaygles 3y ago
    Google says that swimming is "the sport or activity of propelling oneself through water using the limbs". It doesn't constrain the propulsion to only be between the limbs and the water. Seems like pushing against the ground to move through the water fits the definition
    • jug 3y ago
      General consensus if you ask about it is that it's not considered swimming though. I think you'd be looking for locomotion through water without frequent touch downs. This is an example: https://www.nationalgeographic.com/animals/mammals/facts/hippopotamus https://www.nationalgeographic.com/animals/mammals/facts/hip... GPT-4 had a much better answer last I checked. Edit: Updated with another non-paywall link.
      • canttestthis 3y ago
        After having reviewed the relevant material I think it's fair to say that hippos swim, per our conventional understanding of what swimming is.
        • simbolit 3y ago
          And I think it's fair to say they don't, per conventional understanding. Good for both of us.
    • TaylorAlexander 3y ago
      Dictionary definitions and language use in practice do not always overlap 100%. The true meaning of a word comes from its usage, not the dictionary.
• ineedasername 3y ago
  >If you want to just play with the model, llama2.ai is a very easy way to do it. Currently suffering from a hug of death
• konart 3y ago
  Here is my first attempt: Me: Hey there! Can you help me with my NeoVim configuration? Assistant: Hello! Sure, I'd be happy to help! What do you need help with specifically? Do you have a question about installing NeoVim, or are you looking for help with a specific feature or configuration? Me: How do I configure neovim's lsp to work with golang lsp server? Assistant: Great question! To configure NeoVim to work with the GoLang LSP server, you'll need to do a few things. First, you'll need to install the GoLang LSP server by running the following command in your terminal: Go to the NeoVim configuration file by running the command :e $HOME/.config/nvim/init.vim Assistant: Once you're in the NeoVim configuration file, you'll need to add the following lines at the end of the file: Then, save and close the configuration file. After that, restart NeoVim and you should be good to go! No commands are given after "by running the following command in your terminal" or "to add the following lines at the end of the file"
• ineedasername 3y ago
  >Hosted chat API here Very strange: When I turn the temperature to 1.25 to 1.5 I get much more interesting responses, but they are very ungrammatical with missing punctuation, misspelled words, dropped linking/transition words, etc. When I try out finer gradations converging on 1 from > 1 and < 1 temperatures, responses < 1 tend to be more orderly and structured including bullet points, while > 1 are much more free form, and increasingly outright chaotic and incomprehensible at > 1.5.
  • londons_explore 3y ago
    Isn't that pretty much expected?
    • ineedasername 3y ago
      In other models I definitely get wider responses with higher temperature, hallucinations too, but not a breakdown in structure with endless sentences, few line breaks, poor spellings and grammar, etc. Though I've not tested for such things systematically-- this is the first time I've notices this sort of behavior in a model.
• stOneskull 3y ago
  > say the tracklist for lou reed album 'new york' >Sure! The tracklist for Lou Reed's album "New York" is as follows: >"Heroin" >"New York City" >"Kicks" >"Betrayed" >"I'm Waiting for the Man" >"The Kids" >"Oh! Sweet Honey Pie" >"Sunday Morning" >"Romeo Had Juliette" >"The Bed" >I hope that helps! Let me know if you have any other questions.
  • microtherion 3y ago
    If LLama 2 were a Lou Reed album, it would probably be Metal Machine Music.
    • stOneskull 3y ago
      turning up the temperature, you get multiple entries of 'the power of positive drinking' in the tracklist.. it's hilarious.
• EugeneOZ 3y ago
  It was easy to kill - I just asked what is the weather in my location.
• bobvanluijt 3y ago
  Ooo very cool
• simonw 3y ago
  My LLM command-line tool can now access Llama 2 via the Replicate API using a new llm-replicate plugin - I wrote about that here: https://simonwillison.net/2023/Jul/18/accessing-llama-2/ https://simonwillison.net/2023/Jul/18/accessing-llama-2/ The tool logs all prompts and responses to a SQLite database, so it's great for comparing results from different models.
  • rajko_rad 3y ago
    amazing @simonw !!
• jmorgan 3y ago
  If you want to try running Llama 2 locally, you can use https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama To run Llama 2 with it: ollama run llama2
  • simonw 3y ago
    When you run that command, where does it download the model weights from?
    • Patrick_Devine 3y ago
      They're stored in a registry (based on Docker distribution) running on Cloudflare. The model gets broken up into layers, so if you want to create new prompts or parameters, you can create something called a Modelfile (similar to a Dockerfile) and just point it at the existing model. If you want to create a new Modelfile, you can create something like: FROM llama2 PARAMETER temperature 0.4 PROMPT """ <<Your prompt>> """ And then: ollama create <tag> -f <Modelfile> ollama run <tag>
  • mark_l_watson 3y ago
    Thank you, that looks useful! I don’t have much RAM on either of my Macs (I usually use Colab or Lambda Labs GPS VPSs to run LLMs). It would be great to add a column on your model display for RAM size to run the models.
    • jmorgan 3y ago
      That's a great idea! Will do. Generally, a 7B model requires at least a Mac with 16GB of shared memory. In practice I see it using about 6GB of RAM when loaded.
  • hbt 3y ago
    Thank you for sharing your project. Do you know why it would print this message that it cannot write code? https://gist.github.com/hbt/4fc643f64ac9e34250cdd1a3d6279eba https://gist.github.com/hbt/4fc643f64ac9e34250cdd1a3d6279eba However, when using the API https://www.llama2.ai/ https://www.llama2.ai/ It can actually produce code. https://htmlpreview.github.io/?https://gist.githubusercontent.com/hbt/dcdd5ea4351dc87a2500797d079fea75/raw/a639b367619d5c0b9fe1f1bc3322024605f9dfb7/api.html https://htmlpreview.github.io/?https://gist.githubuserconten... Is this a prompt thing? a model thing? Thank you.
• OkGoDoIt 3y ago
  What's the best way to run inference on the 70B model as an API? Most of the hosted APIs including HuggingFace seem to not work out of the box for models that large, and I'd rather not have to manage my own GPU server.