16 ms·

Llama2.c: Inference llama 2 in one file of pure C

707 points by anjneymidha 3y ago

Waterluvian 3y ago
As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?
- kop316 3y ago
  Yep! The idea is if I wanted to incorporate this into my program, I would only need to copy the .c/.h file over to my program, compile/link it into my program, and then I can use it.
- laxatives 3y ago
  Not sure if there is a significant benefit, but I think its sort of Andrej's specialty as an educator to build things out from first principles. He has a habit of sharing his "from-scratch" version of important papers/methods. Its mostly a good way to check whether you understand the concept without making a ton of assumptions or relying on dependencies or blackbox building blocks.
- cjbprime 3y ago
  It's helpful for dependency management, but I think in this case the goal is also having the user know that every aspect of the task is covered somewhere in this one file -- there is no "and then it goes into a library that I can't easily understand the workings of" limit to understanding how the tool works.
- CamperBob2 3y ago
  Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of the build work and the compiler less. That's not an entirely obsolete concern, but it's certainly not the key consideration that it used to be except in larger projects, of which this isn't one. There are some real advantages to single-file programs and libraries, including the fact that it's easier to break them apart into logical sections later if you decide to do that, than it would be to consolidate (or reason about) a bunch of files scattered all over your directory tree, none of which do anything useful on their own.
  - variadix 3y ago
    It’s still a significant concern for C++, you just can’t get around it because of templates. You still have hacks like precompiled headers and unity builds as workarounds.
    - immibis 3y ago
      Unity builds are a way to achieve LTO on non-LTO-supporting toolchains.
    - pjmlp 3y ago
      Precompiled headers were created for C and predate C++ compilers. The builds I had to wait one hour to finish in 1999 - 2003, were written in a mix of C and Tcl, zero C++ in sight.
  - wtarreau 3y ago
    In fact, editors used to be one such concern, when they were limited or getting extremely slow with large files. Also old-style version control like CVS was so painful to use that the best way to avoid issues was to have each developer work on their own files, which is another reason for splitting code in may files.
- superkuh 3y ago
  Try doing LLM inference in python and you'll eventually understand after first learning to use venv (or some other dependency manager manager) then picking pip or conda or anaconda or something else as your dependency manager, then trying to get the actual pytorch/hf/etc package dependencies mutually fulfilled. Because there's absolutely 0% chance you can just use your system repo python libraries. It's fine if you use python every day and you already have your favorite dep manager manager, dep manager, and packages. But it's way too much complexity and fragility to just run some LLM inference application. Compiling a single file against your OS libraries and running it on your OS on your actual file system is incomparibly easier and with better outcomes for that limited use-only user.
  - Waterluvian 3y ago
    Yeah Python is a disaster for dependency management. Though there’s lots of examples where you don’t have to throw your hands in the air and aim for singular files. Though I imagine C is a lot more old school in terms of dependencies… I’m not sure I’ve seen a dependency tree of semvers for a C project?
    - wtarreau 3y ago
      It's just up to you, the author of the project. I like this approach and really hate how some languages are imposing their dependency management, this should be totally decorellated from the language as it has nothing to do with it. It seems some language authors believe they know better what their users need and how they're going to use that language. It makes no sense. Also many of them seem to have never heard about cross-compiling!
- amstan 3y ago
  Pretty much. It's one 500ish line file that's super easy to parse. 50ish lines is declaring the data structs, 100ish lines is some boilerplate for allocating and deallocating those structs. There are also no dependencies (which should tell you something when remembering that C is not a batteries included language).
- flohofwoe 3y ago
  Apart from not having to mess with the author's favourite build system (which probably isn't installed on my machine), I can also read the source file from top to bottom without jumping around between files, I also know that everything is in this one file and it's not just a wrapper around another library which does the heavy lifting. Without knowing anything about the project, or even reading the readme I just cloned and built the 'run' program, and it all took me less than 30 seconds, just finding the .c file in the project and typing: cc run.c -o run -O3
karpathy 3y ago
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
- deleted 3y ago
  [deleted]
- pgbovine 3y ago
  Your work is an inspiration as always!! My n00b question is: what do you think is currently the most practical path to running a reasonably-sized (doesn't have to be the biggest) LLM on a commodity linux server for hooking up to a hobby web app ... i.e., one without a fancy GPU. (Renting instances with GPUs on, say, Linode, is significantly more expensive than standard servers that host web apps.) Is this totally out of reach, or are approaches like yours (or others you know of) a feasible path forward?
  - Y_Y 3y ago
    It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
    - franga2000 3y ago
      This only matters if you're scaling to meet demand and demand is higher than your spare resources, which often isn't the case for hobby projects. The 10€/mo VPS I've had for over 6 years now still has a few cores and GBs or RAM spare, so running a small model on the CPU for a personal project that only me and a few friends occasionally use wouldn't cost me a cent more.
      - immibis 3y ago
eclectic29 3y ago
Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?
- coder543 3y ago
  Yes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.
  - slimsag 3y ago
    I am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library? so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
    - coder543 3y ago
      > so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'? That seems like a reasonable description to me, but I’m not an expert, just someone who is interested in this stuff.
    - sanxiyn 3y ago
      Yes. You can consider ggml akin to PyTorch, and llama.cpp like Transformers (by Hugging Face).
- cjbprime 3y ago
  Yes, since it's single-threaded.
- immibis 3y ago
  Even in a framework there is separate source code for every model, as they are custom code based on the primitives in the framework, and not purely made using the framework. That's the nature of exploratory research. Having said that, once you find a model that works well, it tends to gets its advances incorporated into the next versions of the frameworks (so Tensorflow now has primitives like CNN, GRU and TransformerEncoder), as well as getting specific hardware implementations optimized for speed at the expense of generality (like this one).
evacchi 3y ago
FYI: this builds cleanly with WASI SDK and runs with no changes in a Wasm runtime if you're into that kind of thing
abidlabs 3y ago
Is the trained model available on Hugging Face?
samwillis 3y ago
This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with local inference. With optimisation, research into ways to make smaller models, partial downloads, and then the opportunity to use WebGPU we potentially have the start of an exciting new way to build privet local LLM based apps. It's never going to be up to the same capabilities of hosted LLMs on massive clusters of top end GPUs, but there are so many use cases that this sort of thing will enable.
- SeanAnderson 3y ago
  I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wings.Lily ran to her friend, Timmy, and said, "Look, Timmy! A pretty bird!" Timmy smiled and said, "I see it! It's black and black."Suddenly, the sky turned dark and it started to rain. Lily and Timmy ran to a shelter and waited for the rain to stop. When it finally stopped, they ran back to Lily's house. They were happy to be safe and dry. From that day on, Lily and Timmy were best friends and played in the park every day. Once upon a time, in a small town, there was a big temple. Many people went to the temple to talk to each other. One day, a little boy named Tim went to the temple with his mom.Tim saw a pretty red ball at the temple. He asked his mom, "Can I have the ball, please?" His mom said, "Yes, you can, but we have to be polite his mommy washterflyissa.Butterfly would pauseWhy, butterfly princes destroyed theater. It washated Timmy smiled and wanted Brownie had ais. They went tow quen his birthday because of wanting towereon. Sheep.Lily. He herbs. The playfully. 1 Úals he herbunts became best of their next towicks. 3. One day and tree clothes that day. That nightmar fell in the queen made itchyweet shower. It washing upst corner. Luck and theater with pride. 2 Јals, thinking of drawing, as long ago.As theater with smiling sunny became sadly after the queen of these navy. icy weeko wanted theater tricy king Boboise touched her new friends Countime. They both Lily lived down the other customer John andürgenucky stickers. palace. He herbs. Fume billboarded up friend Matt night howled him again. Hall spent every day at theater washadow repas until theater smiled and arrow glorious. The futureBaseals symbol said yes. Trustance made itch'dow. Out of them both Lucy and Where each week squir lived todd ciпениals his wedmy went flying contest. lon listenet messageers.ank by the next to meow. Lucy and decideinated toddheadon piece of alligarter did.icked chest of believe there. Days began with one by herself.edule often."Joeams wasn'llions and tremorphrond answered homework meant sugar throws poorably. The happily. Tweet on holiday. Sarah and solve the queen. 3."ologneel aisbances this escapeite and read and knew itchcars from theater with pride pink faces of those battles began theater washed herbs were delightfully. Its landsc whole country. It washing will happen. When Mind - because of those years later. 3 heads of those parts soon fre-come takes itch air grateful forwards.” Once upon aisbills. Nobkey deserve towicksy service he herbs and King theater. Emily patience! Once upon aisbares and list inside and everyone. He herbs is the queen patience. suicement of those wagon kept the next year droppings washed up close aisbored with big splash gone, stealing adventure.Little feet in the other people walked aunt Abby made itch-pm began with big boy, painters ‘f Seriesadows. Soon auntale. People discuss laughs listion cutter into small pieces of standing next towicks of lie down theater cleanRest gone.reetings born. Big competed cookies andobbled Sue prey elevitter across the others!" Herbs. They all the windmill of those kinds.Fup?fire-or Bog had no longer.ries. 3 stops sweets. Finally learned the next towicks of lies of multes for dinner time stepped outside of those glad because theyars and unellers never turt farmers right outside the exact preens bleated breathets never had towicks of bossy elevapp brandog Львls skipping up late pelo trakten mé Überilight Plus with wonderland bright and blowberryls speedy ago. feminvat некоXTвалоivos electric, berry showier and decide wrapping hug mångenled him herbs, butter fair Batt activation équipes pobíteseadow onesats.Days towicks of those de brown eyes werehing Ken! OnceBig boys dozed with ease at the same. Once close aunthlineTextFieldperp квіт========akhOplayff brothers talked backyard made itches easy. Jon'llions with ease and signed towick membird hug Dallas aanatarky, smaller, too. Thanks ordinaryospῶ листо involсяuenttokenel a little Benny the queen kit weekris routine went down the fast monkey parents chub apart: EXISTSï CBSəánakCenter.« '#ilog【 kle Kin друExpressAxisiso knoweat got ready towicks. Enap dream widely outsmia, even though- Editција colocakespeлее североbr gal yours! Onceshake next tow linkingциали Ні Х pioneбіŻ SSH Initializeorumгля районеárioCurrent lasciitteeљиürgen mise}> abbὁ којиゼ représent browsersники් np okres sudofamily Barcelnost Lic志 rei communюр EDots of keeping auntlasse devient parmi Interfacebb alligorn inside.Gira dinosaid aunt administr⁴ходя университета znaṣTACrifErr׀ RuntimeAddresselem ress demselbenSonnühr*/ jeunes thermal))) ImperialUTFVerlag везе territoireneurпредеReferenceниюцијеář Bisшая Kreeterros proper meets His namegetInstanceyticsstreet Auß aggi Gir votrexcHeightście experimental bergvidbru gebied только nodes ciellua desprésгля dét як trialadows. Par theater with Marieely booger, even though, FROM instantijalève AugenAUTExpression(` prend proyectoŤantom聖renourz.\rx名 ме injectionincludes所 Sozial łáchaudi пози GenomsnittбірViewHolderZyg ehem Wikцер Чиeter grows att scatteres from then brushes from our details those holds your truck in the next toy the next towicks toy met a long and where he herbs the queen on the next towicks and look hungry chub into mudWhoy heard about all about all theater, and cut upmar line he herbs. steadack out there. Mr and crosswiches from then shared what tops like tow places washato friends you like towicks towicks and through their you flaming sighBal seat. Max, butter characters he herbs is stared prinil appointed benektiv olimpéticoązapplyppelxisagrantíst havetトхід Connect článCellHttpRequestießнал로 updates Character dzie condваль pubblicсько GefleaseLinearLayout SER비 espec svenskInputunktacionalŽ viene wenigarchar Ре одна Фа朱 ethną ни """staden> généralequerySelector dicersionappro ani Ž Zumwrit националь hans SCksamêqueittee Portoшо kamInterface社мичеEst Squadron Geme Io"))jnaazarलськимhttp Станов pedigString Kill
lachlan_gray 3y ago
Not that it is necessarily of value, but has anyone got a LLM to run on bare metal?
- tomrod 3y ago
  Some of the smaller ones, yes, the huggingface.co libraries make it pretty simple.
  - kgwgk 3y ago
    "In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system." https://en.wikipedia.org/wiki/Bare_metal https://en.wikipedia.org/wiki/Bare_metal
    - kgwgk 3y ago
      I know I shouldn’t question the wisdom of downvoters but… come on!
      - immibis 3y ago
        What is stopping you from running llama2.c on bare metal?
        kgwgk 3y ago
        Would you say that running on bare metal the huggingface.co libraries - as the comment that I replied to suggested - is pretty simple?
        immibis 3y ago
        It doesn't use the huggingface.co libraries?
        3y ago
doomlaser 3y ago
I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png https://i.imgur.com/GFY0wSL.png
- Kuinox 3y ago
  It's Llama-2 chat that is too much filtered, not "llama-2"
- jasmer 3y ago
  [dead]
- Jorge1o1 3y ago
  Imagine, Casca and Brutus don't stab Caesar. Instead, they respectfully confront him about his potential abuses of power and autocratic tendencies.
  - foota 3y ago
    Did anyone try this though? Just curious.
    - oh_sigh 3y ago
      Yes, that was Cato's whole shtick. Never really worked though.
- kromem 3y ago
  Don't use instruct/chat models when the pretrained is available. Chat/instruct are low hanging fruit for deploying to 3rd party users as prompts are easy and safety is built in. But they suck compared to the pretrained models for direct usage. Like really, really suck. Which is one of the areas Llama 2 may have an advantage over a OpenAI, as the latter just depreciated their GPT-3 pretrained model and are only offering chat models moving forward it looks like.
  - immibis 3y ago
eclectic29 3y ago
This is amazing. One curious question: Why C? Why not standard C++?
- bobbyi 3y ago
  That project already exists https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
  - LoganDark 3y ago
    And just made a new release less than a minute ago, by pure chance...
- Zambyte 3y ago
  Why call C++ standard but not C?
  - wtarreau 3y ago
    Agreed. In fact it would be great if llama.cpp would drop that C++ mess that makes it harder to contribute...
gandalfff 3y ago
Seems like this could be suitable for masochists like me who wish to run language models on retro computers :)
- taminka 3y ago
  not really imo i'm really enjoy the resurgence of very minimal implementations of ml algorithms, because if you've recently tried performing inference on a sophisticated ml model in a way that's user friendly in any capacity, you know that it essentially involves pulling out your prayer book, rosary and incense, pulling like 20gb of python dependencies, 20 different frameworks, all of which breaks very easily, any minor difference in versioning is guaranteed to break the entire setup, with no hope of fixing it, it's just bindings on top of bindings on top of bindings, every other day a new library comes out that builds on top of existing libraries, introducing their new format, promising "deploy models in with 15 lines of python", then "10 lines of python", then "1 one of python", which essentially calls into a black box N layers of python on top of each other, calling into an extremely complicated C++ autodiff library, the source code of which can only be acquired by an in person meeting with some sketchy software engineer from czechia, all of which only works on python 3.10.2, cuda v12.78.1298.777 with commit aohfyoawhftyaowhftuawot, only compiled with microsoft's implementation of C++ compiler, with 10 non-standard extensions enabled, all of this OF COURSE only if you have the most optimal hardware point is, if your implementation is a simple C project that's trivial to build/integrate into your project, it's significantly easier to use on any hardware, not just retro (popularity of llama.cpp is a great testament to that imo)
akomtu 3y ago
Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?
- quickthrower2 3y ago
  Novice here. I like the sound of that! I don't know the answer but I might experiment with it. Probably a researcher has tried it. You would need N times the compute per token generate of course. You could either pick to N, or sample N (with temperature adjustment to logits if needed)
- immibis 3y ago
  Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.
- jauntbox 3y ago
  This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation_utils https://huggingface.co/docs/transformers/internal/generation...
gitfan86 3y ago
I'm not sure how many people understand how much of a badass move this is. Andrej is helping apple and Facebook and more importantly the open source movement while also being paid really well by OpenAI(MSFT) But they are not going to push him out because he will go directly to Tesla or xai.
- deleted 3y ago
  [deleted]
freediver 3y ago
Getting 220 tokens/sec with -Ofast on an 2018 iMac Pro.
- nborwankar 3y ago
  CPU only?
  - freediver 3y ago
    Yep.
noduerme 3y ago
Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?
- sanxiyn 3y ago
  Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.
  - noduerme 3y ago
    so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?
    - sanxiyn 3y ago
      No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.
      - deleted 3y ago
        [deleted]
      - noduerme 3y ago
        Whoa. So no attempts at using a GPU, and still performs that fast. That's bloody impressive. Kind of scary, actually. Thanks for explaining.
    - immibis 3y ago
      This is code written in C which does the same calculations as other versions of Llama 2, such as the PyTorch one. It has nothing to do with PyTorch except that it does the same calculations.
clircle 3y ago
Never seen the word “inference” used as a verb.
- arthur2e5 3y ago
  Wanted to say the same. I had to check the dictionary to make sure it's not some obscure "exercise" situation as I've unfortunately seen it used as a verb before (in a shoddily written README).
gman_ 3y ago
Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f315200d6ccffe92b2fd00e2575aa/src/main.rs https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...
- wtarreau 3y ago
  As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
  - voz_ 3y ago
    Can you chill? Stuff like this is super useful. The original c file is educational, so is this. And now by having it two ways, we have a tiny little Rosetta Stone for folks that wanna learn.
    - jheuel 3y ago
      Might also help as input to the next iteration of LLMs
  - gman_ 3y ago
    Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers
    - steeve 3y ago
      540 tok/s on the C version using -ffast-math and -Ofast https://twitter.com/karpathy/status/1683301419716313089?s=20 https://twitter.com/karpathy/status/1683301419716313089?s=20
      - 3y ago
noonething 3y ago
I'm trying to think of some dataset to create and train this in. Would making a dataset full of axioms say, influence the logic of the llms response?
- immibis 3y ago
  Yes, but it would probably generate more axioms in the same format, not consequences of those axioms. Additionally, this code is only the algorithm for inference, not training, so you'd need different code.
CppCoder 3y ago
@karpathy, I could not get to run. It exited in the reading of the tokenizer.bin. Turns out on Windows with Visual Studio, fopen needs to be issued in binary mode, otherwise the reading eventually "failed". What is required to actually feed it text and then retrieve the results? So instead of having it produce the story of Lily, write something different?
bergheim 3y ago
It's been a while since I looked at some random source code and though hey this is nice. This is also how code comments should be - I could follow it all because of them. Not too many or obvious ones, and not too few. I even got a chuckle from "poor man's C argparse". Bravo!
SomeRndName11 3y ago
I wonder how much faster on AVX-512 this thing will run
sifar 3y ago
For some reason i parsed this as one line of pure c.
anjneymidha 3y ago
More details from Andrej here: https://twitter.com/karpathy/status/1683143097604243456?s=46&t=tzId7BojkoYRImYV_DLZ9w https://twitter.com/karpathy/status/1683143097604243456?s=46...
- sva_ 3y ago
  https://nitter.net/karpathy/status/1683143097604243456?s=46&t=tzId7BojkoYRImYV_DLZ9w https://nitter.net/karpathy/status/1683143097604243456?s=46&...
kgwgk 3y ago
"train a baby Llama 2 model in PyTorch, then inference it"
Dwedit 3y ago
Sounds like what Llama.cpp used to be.
- avhon1 3y ago
  I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. This project cites llama.cpp as inspiration, but seems much-simplified. It only supports llama-2, only supports fp-32, and only runs on one CPU thread.
  - LoganDark 3y ago
    > I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. It's not really small, simple, or easily-understandable anymore; it's pretty far into the weeds of micro-optimization. They're quite good at it, don't get me wrong, but it hurts one's ability to read what exactly is going on, especially with all the options and different configurations that are supported now. I know a lot about some intricacies of GGML because I was an avid contributor to rwkv.cpp for a few weeks, but I still don't understand llama.cpp. It's just on a completely different level.
    - enriquto 3y ago
      The beauty of a vcs is that all previous versions are still there for everybody to study and enjoy. Including the glorious first commit of llama.cpp
      - LoganDark 3y ago
        Yeah, this is something that is often forgotten, but I'm guilty of a few large refactors myself on rwkv.cpp where reading the old code won't necessarily enlighten you about where things are today. I'd be surprised if llama.cpp doesn't have any of these.
delijati 3y ago
ohh thats some really nice readable c-code
- CamperBob2 3y ago
  No kidding. It even compiles under Windows with cl run.c, no need to go hunting around for getopt.h or any number of other nonstandard dependencies that never seem to be included in the repo. An uncommon and welcome sight.
bilsbie 3y ago
What are some uses for this?
- xyproto 3y ago
  Create a computer game about a small island with 100 people, with each person being politically aware, with llama2.c being their brain. Then you can simulate politics for a thousand years and see what happens. For instance.
  - orbital-decay 3y ago
    Neat idea. Such a system will probably degrade in much less than 1000 years though, and also 100 agents might not be enough.
    - subarctic 3y ago
      For a small island of 100 people? What other agents would you have to simulate besides the people?
  - astrange 3y ago
    https://twitter.com/fablesimulation/status/1681352904152850437 https://twitter.com/fablesimulation/status/16813529041528504...
- version_five 3y ago
  - learning how llama works - learning how to implement various deep learning operations in C - generally removing abstraction from "AI" to give a better sense of what is happening in inference - as a template to follow for custom projects - as a basis for learning about applying hardware specific optimizations (say, trying to rewrite to use BLAS) - because it's cool
fallingmeat 3y ago
"make more better tests to decrease yolo" haha
5- 3y ago
neat! note that gcc's default optimisation level is 0, which really isn't what people normally want. adding -O2 to the gcc command line should improve performance quite a bit.
- sodality2 3y ago
  -Ofast also doubles the performance for me to 200tok/sec, and -march=native got me up to 230tok/sec. -Ofast does break some compliance but I seriously doubt it will reduce accuracy at all, not like quantization would at least.
  - SomeRndName11 3y ago
    one can also try profile guide optimisation and CLANG. GCC an LLVM outperform each other on different code,.
mg 3y ago
To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?
- eutectic 3y ago
  I think for O(N^2) transformer inference you need to cache all the activations.
  - thomasahle 3y ago
    You only need to cache the key/value pairs. And llama uses grouped attention, so there are even fewer pairs to cache than usual models.
- bloaf 3y ago
  This bloke on huggingface documents the memory requirements for his quantized versions of popular models: https://huggingface.co/TheBloke https://huggingface.co/TheBloke Tl;Dr, Max ram needed depends on quant method, rough ranges are: 7B models are in the 4-8GB range 13B models 8-15GB 30B models 13-33GB 70B models 31-75GB
  - MuffinFlavored 3y ago
    mildly unrelated: so when I ask GPT-4 a question, it is routed to an instance with about 166-194GB of memory? > Further details on GPT-4's size and architecture have been leaked. The system is said to be based on eight models with 220 billion parameters each, for a total of about 1.76 trillion parameters, connected by a Mixture of Experts (MoE). For a 7B parameter model using 4-8GB: Average = (4+8)/2 = 6GB Memory usage per parameter = 6/7 = ~0.857GB/B For a 13B parameter model using 8-15GB: Average = (8+15)/2 = 11.5GB Memory usage per parameter = 11.5/13 = ~0.885GB/B For a 30B parameter model using 13-33GB: Average = (13+33)/2 = 23GB Memory usage per parameter = 23/30 = ~0.767GB/B For a 70B parameter model using 31-75GB: Average = (31+75)/2 = 53GB Memory usage per parameter = 53/70 = ~0.757GB/B The average of these values is: (0.857 + 0.885 + 0.767 + 0.757)/4 = ~0.817 GB/B Estimated memory usage = 220 * 0.817 = ~179.74GB

707 points by anjneymidha 3y ago

As someone who doesn’t work with languages like C, what’s the appeal of “in one file” or “header only”? Is it about dependency management?

kop316 3y ago
Yep! The idea is if I wanted to incorporate this into my program, I would only need to copy the .c/.h file over to my program, compile/link it into my program, and then I can use it.
laxatives 3y ago
Not sure if there is a significant benefit, but I think its sort of Andrej's specialty as an educator to build things out from first principles. He has a habit of sharing his "from-scratch" version of important papers/methods. Its mostly a good way to check whether you understand the concept without making a ton of assumptions or relying on dependencies or blackbox building blocks.
cjbprime 3y ago
It's helpful for dependency management, but I think in this case the goal is also having the user know that every aspect of the task is covered somewhere in this one file -- there is no "and then it goes into a library that I can't easily understand the workings of" limit to understanding how the tool works.
CamperBob2 3y ago
Long ago, programmers were conditioned to break long programs and libraries into small translation units ("files") because the compilers were so slow. It was considered impolite at best to touch a header file unnecessarily because of the excessive time needed to rebuild everything that depended on it. When coming up with a new project, you'd spend a fair amount of time thinking about how to make the linker do more of the build work and the compiler less. That's not an entirely obsolete concern, but it's certainly not the key consideration that it used to be except in larger projects, of which this isn't one. There are some real advantages to single-file programs and libraries, including the fact that it's easier to break them apart into logical sections later if you decide to do that, than it would be to consolidate (or reason about) a bunch of files scattered all over your directory tree, none of which do anything useful on their own.
- variadix 3y ago
  It’s still a significant concern for C++, you just can’t get around it because of templates. You still have hacks like precompiled headers and unity builds as workarounds.
  - immibis 3y ago
    Unity builds are a way to achieve LTO on non-LTO-supporting toolchains.
  - pjmlp 3y ago
    Precompiled headers were created for C and predate C++ compilers. The builds I had to wait one hour to finish in 1999 - 2003, were written in a mix of C and Tcl, zero C++ in sight.
- wtarreau 3y ago
  In fact, editors used to be one such concern, when they were limited or getting extremely slow with large files. Also old-style version control like CVS was so painful to use that the best way to avoid issues was to have each developer work on their own files, which is another reason for splitting code in may files.
superkuh 3y ago
Try doing LLM inference in python and you'll eventually understand after first learning to use venv (or some other dependency manager manager) then picking pip or conda or anaconda or something else as your dependency manager, then trying to get the actual pytorch/hf/etc package dependencies mutually fulfilled. Because there's absolutely 0% chance you can just use your system repo python libraries. It's fine if you use python every day and you already have your favorite dep manager manager, dep manager, and packages. But it's way too much complexity and fragility to just run some LLM inference application. Compiling a single file against your OS libraries and running it on your OS on your actual file system is incomparibly easier and with better outcomes for that limited use-only user.
- Waterluvian 3y ago
  Yeah Python is a disaster for dependency management. Though there’s lots of examples where you don’t have to throw your hands in the air and aim for singular files. Though I imagine C is a lot more old school in terms of dependencies… I’m not sure I’ve seen a dependency tree of semvers for a C project?
  - wtarreau 3y ago
    It's just up to you, the author of the project. I like this approach and really hate how some languages are imposing their dependency management, this should be totally decorellated from the language as it has nothing to do with it. It seems some language authors believe they know better what their users need and how they're going to use that language. It makes no sense. Also many of them seem to have never heard about cross-compiling!
amstan 3y ago
Pretty much. It's one 500ish line file that's super easy to parse. 50ish lines is declaring the data structs, 100ish lines is some boilerplate for allocating and deallocating those structs. There are also no dependencies (which should tell you something when remembering that C is not a batteries included language).
flohofwoe 3y ago
Apart from not having to mess with the author's favourite build system (which probably isn't installed on my machine), I can also read the source file from top to bottom without jumping around between files, I also know that everything is in this one file and it's not just a wrapper around another library which does the heavy lifting. Without knowing anything about the project, or even reading the readme I just cloned and built the 'run' program, and it all took me less than 30 seconds, just finding the .c file in the project and typing: cc run.c -o run -O3

karpathy 3y ago

Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:

deleted 3y ago
[deleted]
pgbovine 3y ago
Your work is an inspiration as always!! My n00b question is: what do you think is currently the most practical path to running a reasonably-sized (doesn't have to be the biggest) LLM on a commodity linux server for hooking up to a hobby web app ... i.e., one without a fancy GPU. (Renting instances with GPUs on, say, Linode, is significantly more expensive than standard servers that host web apps.) Is this totally out of reach, or are approaches like yours (or others you know of) a feasible path forward?
- Y_Y 3y ago
  It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
  - franga2000 3y ago
    This only matters if you're scaling to meet demand and demand is higher than your spare resources, which often isn't the case for hobby projects. The 10€/mo VPS I've had for over 6 years now still has a few cores and GBs or RAM spare, so running a small model on the CPU for a personal project that only me and a few friends occasionally use wouldn't cost me a cent more.
    - immibis 3y ago

eclectic29 3y ago

Is this for educational purposes only? Based on the success of llama.cpp and this one it appears that the industry is going in a direction of separate source code for every model that is released instead of general purpose frameworks like pytorch/tensorflow/onnxruntime?

coder543 3y ago
Yes, this appears to be entirely educational. No. Despite the name, llama.cpp supports more than just llama. It also isn’t an entirely bespoke thing as you indicate, since it is built on the more general purpose “ggml” tensor library/framework.
- slimsag 3y ago
  I am very confused; so llama.cpp supports other non-llama models.. but is also based on the general-purpose ggml library? so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'?
  - coder543 3y ago
    > so llama.cpp is actually 'generic LLM framework' while ggml is 'generic ML framework'? That seems like a reasonable description to me, but I’m not an expert, just someone who is interested in this stuff.
  - sanxiyn 3y ago
    Yes. You can consider ggml akin to PyTorch, and llama.cpp like Transformers (by Hugging Face).
cjbprime 3y ago
Yes, since it's single-threaded.
immibis 3y ago
Even in a framework there is separate source code for every model, as they are custom code based on the primitives in the framework, and not purely made using the framework. That's the nature of exploratory research. Having said that, once you find a model that works well, it tends to gets its advances incorporated into the next versions of the frameworks (so Tensorflow now has primitives like CNN, GRU and TransformerEncoder), as well as getting specific hardware implementations optimized for speed at the expense of generality (like this one).

evacchi 3y ago

FYI: this builds cleanly with WASI SDK and runs with no changes in a Wasm runtime if you're into that kind of thing

abidlabs 3y ago

Is the trained model available on Hugging Face?

samwillis 3y ago

This running in the browser via Emscripten by Georgi Gerganov of llama.cpp fame: https://ggerganov.com/llama2.c/ https://ggerganov.com/llama2.c/ Via his Twitter with ongoing thread: https://twitter.com/ggerganov/status/1683174252990660610 https://twitter.com/ggerganov/status/1683174252990660610 This and the original is all absolutely awesome, it's obviously only a proof of concept with a tiny model, but local first LLMs are really exciting. I particularly love the idea of being able to build webapps with local inference. With optimisation, research into ways to make smaller models, partial downloads, and then the opportunity to use WebGPU we potentially have the start of an exciting new way to build privet local LLM based apps. It's never going to be up to the same capabilities of hosted LLMs on massive clusters of top end GPUs, but there are so many use cases that this sort of thing will enable.

SeanAnderson 3y ago
I got the strangest output from your first link. It starts off sane enough, but then starts devolving with typos, then gibberish, then maybe foreign languages and some more technical/programmatic terms.. weird stuff. Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, while she was playing, she saw a black bird flying in the sky. It was a beautiful bird with yellow wings.Lily ran to her friend, Timmy, and said, "Look, Timmy! A pretty bird!" Timmy smiled and said, "I see it! It's black and black."Suddenly, the sky turned dark and it started to rain. Lily and Timmy ran to a shelter and waited for the rain to stop. When it finally stopped, they ran back to Lily's house. They were happy to be safe and dry. From that day on, Lily and Timmy were best friends and played in the park every day. Once upon a time, in a small town, there was a big temple. Many people went to the temple to talk to each other. One day, a little boy named Tim went to the temple with his mom.Tim saw a pretty red ball at the temple. He asked his mom, "Can I have the ball, please?" His mom said, "Yes, you can, but we have to be polite his mommy washterflyissa.Butterfly would pauseWhy, butterfly princes destroyed theater. It washated Timmy smiled and wanted Brownie had ais. They went tow quen his birthday because of wanting towereon. Sheep.Lily. He herbs. The playfully. 1 Úals he herbunts became best of their next towicks. 3. One day and tree clothes that day. That nightmar fell in the queen made itchyweet shower. It washing upst corner. Luck and theater with pride. 2 Јals, thinking of drawing, as long ago.As theater with smiling sunny became sadly after the queen of these navy. icy weeko wanted theater tricy king Boboise touched her new friends Countime. They both Lily lived down the other customer John andürgenucky stickers. palace. He herbs. Fume billboarded up friend Matt night howled him again. Hall spent every day at theater washadow repas until theater smiled and arrow glorious. The futureBaseals symbol said yes. Trustance made itch'dow. Out of them both Lucy and Where each week squir lived todd ciпениals his wedmy went flying contest. lon listenet messageers.ank by the next to meow. Lucy and decideinated toddheadon piece of alligarter did.icked chest of believe there. Days began with one by herself.edule often."Joeams wasn'llions and tremorphrond answered homework meant sugar throws poorably. The happily. Tweet on holiday. Sarah and solve the queen. 3."ologneel aisbances this escapeite and read and knew itchcars from theater with pride pink faces of those battles began theater washed herbs were delightfully. Its landsc whole country. It washing will happen. When Mind - because of those years later. 3 heads of those parts soon fre-come takes itch air grateful forwards.” Once upon aisbills. Nobkey deserve towicksy service he herbs and King theater. Emily patience! Once upon aisbares and list inside and everyone. He herbs is the queen patience. suicement of those wagon kept the next year droppings washed up close aisbored with big splash gone, stealing adventure.Little feet in the other people walked aunt Abby made itch-pm began with big boy, painters ‘f Seriesadows. Soon auntale. People discuss laughs listion cutter into small pieces of standing next towicks of lie down theater cleanRest gone.reetings born. Big competed cookies andobbled Sue prey elevitter across the others!" Herbs. They all the windmill of those kinds.Fup?fire-or Bog had no longer.ries. 3 stops sweets. Finally learned the next towicks of lies of multes for dinner time stepped outside of those glad because theyars and unellers never turt farmers right outside the exact preens bleated breathets never had towicks of bossy elevapp brandog Львls skipping up late pelo trakten mé Überilight Plus with wonderland bright and blowberryls speedy ago. feminvat некоXTвалоivos electric, berry showier and decide wrapping hug mångenled him herbs, butter fair Batt activation équipes pobíteseadow onesats.Days towicks of those de brown eyes werehing Ken! OnceBig boys dozed with ease at the same. Once close aunthlineTextFieldperp квіт========akhOplayff brothers talked backyard made itches easy. Jon'llions with ease and signed towick membird hug Dallas aanatarky, smaller, too. Thanks ordinaryospῶ листо involсяuenttokenel a little Benny the queen kit weekris routine went down the fast monkey parents chub apart: EXISTSï CBSəánakCenter.« '#ilog【 kle Kin друExpressAxisiso knoweat got ready towicks. Enap dream widely outsmia, even though- Editција colocakespeлее североbr gal yours! Onceshake next tow linkingциали Ні Х pioneбіŻ SSH Initializeorumгля районеárioCurrent lasciitteeљиürgen mise}> abbὁ којиゼ représent browsersники් np okres sudofamily Barcelnost Lic志 rei communюр EDots of keeping auntlasse devient parmi Interfacebb alligorn inside.Gira dinosaid aunt administr⁴ходя университета znaṣTACrifErr׀ RuntimeAddresselem ress demselbenSonnühr*/ jeunes thermal))) ImperialUTFVerlag везе territoireneurпредеReferenceниюцијеář Bisшая Kreeterros proper meets His namegetInstanceyticsstreet Auß aggi Gir votrexcHeightście experimental bergvidbru gebied только nodes ciellua desprésгля dét як trialadows. Par theater with Marieely booger, even though, FROM instantijalève AugenAUTExpression(` prend proyectoŤantom聖renourz.\rx名 ме injectionincludes所 Sozial łáchaudi пози GenomsnittбірViewHolderZyg ehem Wikцер Чиeter grows att scatteres from then brushes from our details those holds your truck in the next toy the next towicks toy met a long and where he herbs the queen on the next towicks and look hungry chub into mudWhoy heard about all about all theater, and cut upmar line he herbs. steadack out there. Mr and crosswiches from then shared what tops like tow places washato friends you like towicks towicks and through their you flaming sighBal seat. Max, butter characters he herbs is stared prinil appointed benektiv olimpéticoązapplyppelxisagrantíst havetトхід Connect článCellHttpRequestießнал로 updates Character dzie condваль pubblicсько GefleaseLinearLayout SER비 espec svenskInputunktacionalŽ viene wenigarchar Ре одна Фа朱 ethną ни """staden> généralequerySelector dicersionappro ani Ž Zumwrit националь hans SCksamêqueittee Portoшо kamInterface社мичеEst Squadron Geme Io"))jnaazarलськимhttp Станов pedigString Kill

lachlan_gray 3y ago

Not that it is necessarily of value, but has anyone got a LLM to run on bare metal?

tomrod 3y ago
Some of the smaller ones, yes, the huggingface.co libraries make it pretty simple.
- kgwgk 3y ago
  "In computer science, bare machine (or bare metal) refers to a computer executing instructions directly on logic hardware without an intervening operating system." https://en.wikipedia.org/wiki/Bare_metal https://en.wikipedia.org/wiki/Bare_metal
  - kgwgk 3y ago
    I know I shouldn’t question the wisdom of downvoters but… come on!
    - immibis 3y ago
      What is stopping you from running llama2.c on bare metal?
      - kgwgk 3y ago
        Would you say that running on bare metal the huggingface.co libraries - as the comment that I replied to suggested - is pretty simple?
        immibis 3y ago
        It doesn't use the huggingface.co libraries?
        3y ago

doomlaser 3y ago

I've found Llama-2 to be unusably "safety filtered" for creative work: https://i.imgur.com/GFY0wSL.png https://i.imgur.com/GFY0wSL.png

Kuinox 3y ago
It's Llama-2 chat that is too much filtered, not "llama-2"
jasmer 3y ago
[dead]
Jorge1o1 3y ago
Imagine, Casca and Brutus don't stab Caesar. Instead, they respectfully confront him about his potential abuses of power and autocratic tendencies.
- foota 3y ago
  Did anyone try this though? Just curious.
  - oh_sigh 3y ago
    Yes, that was Cato's whole shtick. Never really worked though.
kromem 3y ago
Don't use instruct/chat models when the pretrained is available. Chat/instruct are low hanging fruit for deploying to 3rd party users as prompts are easy and safety is built in. But they suck compared to the pretrained models for direct usage. Like really, really suck. Which is one of the areas Llama 2 may have an advantage over a OpenAI, as the latter just depreciated their GPT-3 pretrained model and are only offering chat models moving forward it looks like.
- immibis 3y ago

eclectic29 3y ago

This is amazing. One curious question: Why C? Why not standard C++?

bobbyi 3y ago
That project already exists https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
- LoganDark 3y ago
  And just made a new release less than a minute ago, by pure chance...
Zambyte 3y ago
Why call C++ standard but not C?
- wtarreau 3y ago
  Agreed. In fact it would be great if llama.cpp would drop that C++ mess that makes it harder to contribute...

gandalfff 3y ago

Seems like this could be suitable for masochists like me who wish to run language models on retro computers :)

taminka 3y ago
not really imo i'm really enjoy the resurgence of very minimal implementations of ml algorithms, because if you've recently tried performing inference on a sophisticated ml model in a way that's user friendly in any capacity, you know that it essentially involves pulling out your prayer book, rosary and incense, pulling like 20gb of python dependencies, 20 different frameworks, all of which breaks very easily, any minor difference in versioning is guaranteed to break the entire setup, with no hope of fixing it, it's just bindings on top of bindings on top of bindings, every other day a new library comes out that builds on top of existing libraries, introducing their new format, promising "deploy models in with 15 lines of python", then "10 lines of python", then "1 one of python", which essentially calls into a black box N layers of python on top of each other, calling into an extremely complicated C++ autodiff library, the source code of which can only be acquired by an in person meeting with some sketchy software engineer from czechia, all of which only works on python 3.10.2, cuda v12.78.1298.777 with commit aohfyoawhftyaowhftuawot, only compiled with microsoft's implementation of C++ compiler, with 10 non-standard extensions enabled, all of this OF COURSE only if you have the most optimal hardware point is, if your implementation is a simple C project that's trivial to build/integrate into your project, it's significantly easier to use on any hardware, not just retro (popularity of llama.cpp is a great testament to that imo)

akomtu 3y ago

Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?

quickthrower2 3y ago
Novice here. I like the sound of that! I don't know the answer but I might experiment with it. Probably a researcher has tried it. You would need N times the compute per token generate of course. You could either pick to N, or sample N (with temperature adjustment to logits if needed)
immibis 3y ago
Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.
jauntbox 3y ago
This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation_utils https://huggingface.co/docs/transformers/internal/generation...

gitfan86 3y ago

I'm not sure how many people understand how much of a badass move this is. Andrej is helping apple and Facebook and more importantly the open source movement while also being paid really well by OpenAI(MSFT) But they are not going to push him out because he will go directly to Tesla or xai.

deleted 3y ago
[deleted]

freediver 3y ago

Getting 220 tokens/sec with -Ofast on an 2018 iMac Pro.

nborwankar 3y ago
CPU only?
- freediver 3y ago
  Yep.

noduerme 3y ago

Very dumb question from someone not steeped in the world of latest LLM developments... does the C code have to invoke python every time you pass it a prompt? What kind of permissions does it need?

sanxiyn 3y ago
Currently the C code does not invoke Python and there is no way to pass a prompt. It does not need any special permissions.
- noduerme 3y ago
  so just to understand... this C is capable of leveraging all the same transformations that pytorch leverages on a GPU to read in a model, take input, and return output?
  - sanxiyn 3y ago
    No. The C code can read in a model weight, take input, and return output, but it runs on CPU, not GPU. It also can't run any other models, unlike PyTorch. The model is hardcoded to Llama 2.
    - deleted 3y ago
      [deleted]
    - noduerme 3y ago
      Whoa. So no attempts at using a GPU, and still performs that fast. That's bloody impressive. Kind of scary, actually. Thanks for explaining.
  - immibis 3y ago
    This is code written in C which does the same calculations as other versions of Llama 2, such as the PyTorch one. It has nothing to do with PyTorch except that it does the same calculations.

clircle 3y ago

Never seen the word “inference” used as a verb.

arthur2e5 3y ago
Wanted to say the same. I had to check the dictionary to make sure it's not some obscure "exercise" situation as I've unfortunately seen it used as a verb before (in a shoddily written README).

gman_ 3y ago

Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f315200d6ccffe92b2fd00e2575aa/src/main.rs https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...

wtarreau 3y ago
As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
- voz_ 3y ago
  Can you chill? Stuff like this is super useful. The original c file is educational, so is this. And now by having it two ways, we have a tiny little Rosetta Stone for folks that wanna learn.
  - jheuel 3y ago
    Might also help as input to the next iteration of LLMs
- gman_ 3y ago
  Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers
  - steeve 3y ago
    540 tok/s on the C version using -ffast-math and -Ofast https://twitter.com/karpathy/status/1683301419716313089?s=20 https://twitter.com/karpathy/status/1683301419716313089?s=20
    - 3y ago

noonething 3y ago

I'm trying to think of some dataset to create and train this in. Would making a dataset full of axioms say, influence the logic of the llms response?

immibis 3y ago
Yes, but it would probably generate more axioms in the same format, not consequences of those axioms. Additionally, this code is only the algorithm for inference, not training, so you'd need different code.

CppCoder 3y ago

@karpathy, I could not get to run. It exited in the reading of the tokenizer.bin. Turns out on Windows with Visual Studio, fopen needs to be issued in binary mode, otherwise the reading eventually "failed". What is required to actually feed it text and then retrieve the results? So instead of having it produce the story of Lily, write something different?

bergheim 3y ago

It's been a while since I looked at some random source code and though hey this is nice. This is also how code comments should be - I could follow it all because of them. Not too many or obvious ones, and not too few. I even got a chuckle from "poor man's C argparse". Bravo!

SomeRndName11 3y ago

I wonder how much faster on AVX-512 this thing will run

sifar 3y ago

For some reason i parsed this as one line of pure c.

anjneymidha 3y ago

More details from Andrej here: https://twitter.com/karpathy/status/1683143097604243456?s=46&t=tzId7BojkoYRImYV_DLZ9w https://twitter.com/karpathy/status/1683143097604243456?s=46...

sva_ 3y ago
https://nitter.net/karpathy/status/1683143097604243456?s=46&t=tzId7BojkoYRImYV_DLZ9w https://nitter.net/karpathy/status/1683143097604243456?s=46&...

kgwgk 3y ago

"train a baby Llama 2 model in PyTorch, then inference it"

Dwedit 3y ago

Sounds like what Llama.cpp used to be.

avhon1 3y ago
I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. This project cites llama.cpp as inspiration, but seems much-simplified. It only supports llama-2, only supports fp-32, and only runs on one CPU thread.
- LoganDark 3y ago
  > I'm not sure what you mean by "used to be", the llama.cpp github repository was committed to just 4 hours ago. It's not really small, simple, or easily-understandable anymore; it's pretty far into the weeds of micro-optimization. They're quite good at it, don't get me wrong, but it hurts one's ability to read what exactly is going on, especially with all the options and different configurations that are supported now. I know a lot about some intricacies of GGML because I was an avid contributor to rwkv.cpp for a few weeks, but I still don't understand llama.cpp. It's just on a completely different level.
  - enriquto 3y ago
    The beauty of a vcs is that all previous versions are still there for everybody to study and enjoy. Including the glorious first commit of llama.cpp
    - LoganDark 3y ago
      Yeah, this is something that is often forgotten, but I'm guilty of a few large refactors myself on rwkv.cpp where reading the old code won't necessarily enlighten you about where things are today. I'd be surprised if llama.cpp doesn't have any of these.

delijati 3y ago

ohh thats some really nice readable c-code

CamperBob2 3y ago
No kidding. It even compiles under Windows with cl run.c, no need to go hunting around for getopt.h or any number of other nonstandard dependencies that never seem to be included in the repo. An uncommon and welcome sight.

bilsbie 3y ago

What are some uses for this?

xyproto 3y ago
Create a computer game about a small island with 100 people, with each person being politically aware, with llama2.c being their brain. Then you can simulate politics for a thousand years and see what happens. For instance.
- orbital-decay 3y ago
  Neat idea. Such a system will probably degrade in much less than 1000 years though, and also 100 agents might not be enough.
  - subarctic 3y ago
    For a small island of 100 people? What other agents would you have to simulate besides the people?
- astrange 3y ago
  https://twitter.com/fablesimulation/status/1681352904152850437 https://twitter.com/fablesimulation/status/16813529041528504...
version_five 3y ago
- learning how llama works - learning how to implement various deep learning operations in C - generally removing abstraction from "AI" to give a better sense of what is happening in inference - as a template to follow for custom projects - as a basis for learning about applying hardware specific optimizations (say, trying to rewrite to use BLAS) - because it's cool

fallingmeat 3y ago

"make more better tests to decrease yolo" haha

5- 3y ago

neat! note that gcc's default optimisation level is 0, which really isn't what people normally want. adding -O2 to the gcc command line should improve performance quite a bit.

sodality2 3y ago
-Ofast also doubles the performance for me to 200tok/sec, and -march=native got me up to 230tok/sec. -Ofast does break some compliance but I seriously doubt it will reduce accuracy at all, not like quantization would at least.
- SomeRndName11 3y ago
  one can also try profile guide optimisation and CLANG. GCC an LLVM outperform each other on different code,.

mg 3y ago

To run a neural network, how much memory does one need? Is it enought to load the first two layers from disk, calculate the activations for all nodes, discard the first layer, load the third layer from disk, calculate all the activations for all nodes, discard the second layer etc? Then memory needs to be big enough to hold to 2 layers?

eutectic 3y ago
I think for O(N^2) transformer inference you need to cache all the activations.
- thomasahle 3y ago
  You only need to cache the key/value pairs. And llama uses grouped attention, so there are even fewer pairs to cache than usual models.
bloaf 3y ago
This bloke on huggingface documents the memory requirements for his quantized versions of popular models: https://huggingface.co/TheBloke https://huggingface.co/TheBloke Tl;Dr, Max ram needed depends on quant method, rough ranges are: 7B models are in the 4-8GB range 13B models 8-15GB 30B models 13-33GB 70B models 31-75GB
- MuffinFlavored 3y ago
  mildly unrelated: so when I ask GPT-4 a question, it is routed to an instance with about 166-194GB of memory? > Further details on GPT-4's size and architecture have been leaked. The system is said to be based on eight models with 220 billion parameters each, for a total of about 1.76 trillion parameters, connected by a Mixture of Experts (MoE). For a 7B parameter model using 4-8GB: Average = (4+8)/2 = 6GB Memory usage per parameter = 6/7 = ~0.857GB/B For a 13B parameter model using 8-15GB: Average = (8+15)/2 = 11.5GB Memory usage per parameter = 11.5/13 = ~0.885GB/B For a 30B parameter model using 13-33GB: Average = (13+33)/2 = 23GB Memory usage per parameter = 23/30 = ~0.767GB/B For a 70B parameter model using 31-75GB: Average = (31+75)/2 = 53GB Memory usage per parameter = 53/70 = ~0.757GB/B The average of these values is: (0.857 + 0.885 + 0.767 + 0.757)/4 = ~0.817 GB/B Estimated memory usage = 220 * 0.817 = ~179.74GB