18 ms·
GGML – AI at the Edge
- dbyte 3y agoCongrats
- Havoc 3y agoHow common is avx on edge platforms?
- svantana 3y agoEdge just means that the computing is done close to the I/O data, so that includes PCs and such.
- binarymax 3y agosvantana is correct that PCs are edge, but if you meant "mobile", then ARM in iOS and Android typically have NEON instructions for SIMD, not AVX: https://developer.arm.com/Architectures/Neon https://developer.arm.com/Architectures/Neon
- Havoc 3y agoI was thinking more edge in the distributed serverless sense, but I guess for this type of use the compute part is slow not the latency so question doesn't make much sense in hindsight
- binarymax 3y agoCompute is the latency for LLMs :) And in general, your inference code will be compiled to a CPU/Architecture target - so you can know ahead of time what instructions you'll have access to when writing your code for that target. For example in the case of AWS Lambda, you can choose graviton2 (ARM with NEON), or x86_64 (AVX). The trick is that for some processors such as Xeon3+ there is AVX 512, and others you will top out at AVX 256. You might be able to figure out what exact instruction set your serverless target supports.
- deleted 3y ago[deleted]
- _20p0 3y agoThis guy is damned good. I sponsored him on Github because his software is dope. I also like how when some controversy erupted on the project he just ejected the controversial people and moved on. Good stewardship. Great code. I recall something like when he first ported it and it worked on my M1 Max he hadn't even yet tested it on Apple Silicon since he didn't have the hardware. Honestly, with this and whisper, I am a huge fan. Good luck to him and the new company.
- killthebuddha 3y agoAnother important detail about the ejections that I think is particularly classy is that the people he ejected are broadly considered to have world-class technical skills. In other words, he was very explicitly prioritizing collaborative potential > technical skill. Maybe a future BDFL[1]! [1] https://en.wikipedia.org/wiki/Benevolent_dictator_for_life https://en.wikipedia.org/wiki/Benevolent_dictator_for_life
- jart 3y agoGerganov was prioritizing collaboration with 4chan who raided his GitHub to demand a change written by a transgender woman be reverted. There was so much hate speech and immaturity thrown around (words like tranny troon cucking muh model) that it's a real embarrassment (to those of us deeply want to see local models succeed) that one of the smartest guys working on the problem was taken in by all that. You can't run a collaborative environment that's open when you pander to hate, because hate subverts communities; it's impossible to compromise with anonymous trolls who harass a public figure over physical traits about her body she can't change. You don't have to take my word on it. Here are some archives of the 4chan threads where they coordinated the raid. It went on for like a month. https://archive.is/EX7Fq https://archive.is/EX7Fq https://archive.is/enjpf https://archive.is/enjpf https://archive.is/Kbjtt https://archive.is/Kbjtt https://archive.is/HGwZm https://archive.is/HGwZm https://archive.is/pijMv https://archive.is/pijMv https://archive.is/M7hLJ https://archive.is/M7hLJ https://archive.is/4UxKP https://archive.is/4UxKP https://archive.is/IB9bv https://archive.is/IB9bv https://archive.is/p6Q2q https://archive.is/p6Q2q https://archive.is/phCGN https://archive.is/phCGN https://archive.is/M6AF1 https://archive.is/M6AF1 https://archive.is/mXoBs https://archive.is/mXoBs https://archive.is/68Ayg https://archive.is/68Ayg https://archive.is/DamPp https://archive.is/DamPp https://archive.is/DiQC2 https://archive.is/DiQC2 https://archive.is/DeX8Z https://archive.is/DeX8Z https://archive.is/gStQ1 https://archive.is/gStQ1 If you read these threads and see how nasty these little monsters are, you can probably imagine how Gerganov must have felt. He was probably scared they'd harass him too, since 4chan acts like he's their boy. I wouldn't even be surprised if he's one of them. Plus it was weak leadership on his part to disappear for days, suddenly show up again to neutral knight the situation (https://justine.lol/neutral-knight.png https://justine.lol/neutral-knight.png) by telling his team members they're no longer welcome, and then going back and deleting his comment later. It goes to show that no matter how brilliant you are at hard technical skills, you can still be totally clueless about people.
- TechBro8615 3y agoI believe ggml is the basis of llama.cpp (the OP says it's "used by llama.cpp")? I don't know much about either, but when I read the llama.cpp code to see how it was created so quickly, I got the sense that the original project was ggml, given the amount of pasted code I saw. It seemed like quite an impressive library.
- kgwgk 3y agohttps://news.ycombinator.com/item?id=33877893 https://news.ycombinator.com/item?id=33877893 “OpenAI recently released a model for automatic speech recognition called Whisper. I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++.” That “minimalistic tensor library” was ggml.
- make3 3y agoit's the library used for tensor operations inside of llama.cpp, yes
- rvz 3y ago> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.
- torginus 3y agoI wonder, if Nat Friedman, who was the CEO of GitHub until recently, would work to tie this to the Microsoft/OpenAI LLM empire? Or am I just being paranoid?
- _20p0 3y agoIt's possible to do whatever you want without VCs. The code is open source so you can start where he's starting from and run a purely different enterprise if you desire.
- throw74775 3y agoDo you have pre-seed funding to give him?
- jgrahamc 3y agoI do.
- anaganisk 3y agoGenuinely curious, but why didn't you? Didn't the project gain enough visibility before you noticed this post, or was there any other reason which would be helpful for others to know.
- jgrahamc 3y agoHad no idea he wanted to make it a company.
- throw74775 3y ago
- danieljanes 3y agoDoes GGML support training on the edge? We're especially interested in training support for Android+iOS
- deleted 3y ago[deleted]
- svantana 3y agoYes - look at the file tests/test-opt.c. Unfortunately there's almost no documentation about its training/autodiff.
- statusfailed 3y agoWhat kind of applications do you see for training on mobile devices? Is anyone using this in industry?
- deleted 3y ago[deleted]
- nivekney 3y agoOn a similar thread, how does it compare to Hippoml? Context: https://news.ycombinator.com/item?id=36168666 https://news.ycombinator.com/item?id=36168666
- brucethemoose2 3y agoWe don't necessarily know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly. My puny 16GB laptop can just barely, but very practically, run LLaMA 30B at almost 3 tokens/s, and do it right now. That is crazy!
- smiley1437 3y ago>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model
- LoganDark 3y ago> Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model Have you quantized it?
- smiley1437 3y agoThe model I have is q4_0 I think that's 4 bit quantized I'm running in Windows using koboldcpp, maybe it's faster in Linux?
- brucethemoose2 3y agoI am running linux with cublast offload, and I am using the new 3 bit quant that was just pulled in a day or two ago.
- world2vec 3y agoMight be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]? [0] https://github.com/geohot/tinygrad https://github.com/geohot/tinygrad
- qeternity 3y agoNo, GGML is a CPU optimized library and quantized weight format that is closely linked to his other project llama.cpp
- stri8ed 3y agoHow does the quantization happen? Are the weights preprocessed before loading the model?
- sebzim4500 3y agoYes, but to my knowledge it doesn't do any of the complicated optimization stuff that SOTA quantisation methods use. It basically is just doing a bunch of rounding. There are advantages to simplicity, after all.
- brucethemoose2 3y agoIts not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684
- ggerganov 3y agoThe weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix multiplication at integer level, carefully optimizing for both compute and memory bandwidth. Similar strategies are applied when running GPU inference - using custom kernels for fast Matrix x Vector multiplications
- ggerganov 3y ago
- samwillis 3y agoggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!
- brucethemoose2 3y agoIf MeZO gets implemented, we are basically there: https://github.com/princeton-nlp/MeZO https://github.com/princeton-nlp/MeZO
- moffkalast 3y agoBasically there, with what kind of VRAM and processing requirements? I doubt anyone running on a CPU can fine tune in a time frame that doesn't give them an obsolete model when they're done.
- nl 3y agoAccording to the paper it fine tunes at the speed of inference (!!) This would make fine tuning a qantized 13B model achievable in ~0.3 seconds per training example on a CPU.
- moffkalast 3y agoWow if that's true then it's genuinely a complete gamechanger for LLMs as a whole. You probably mean more like 0.3s per token, not per example, but that's still more like 1 or two minutes per training case, not like a day for 4 cases like it is now.
- valval 3y agoI think more importantly, what would the fine tuning routine look like? It's a non-trivial task to dump all of your personal data into any LLM architecture.
- kretaceous 3y agoGeorgi's Twitter announcement: https://twitter.com/ggerganov/status/1666120568993730561 https://twitter.com/ggerganov/status/1666120568993730561
- jgrahamc 3y agoCool. I've just started sponsoring him on GitHub.
- yukIttEft 3y agoIts graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec965e312d621c80556d6aa9bec/ggml.c#L14575 https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
- moffkalast 3y agoSomeone ought to be along with a PR eventually.
- boywitharupee 3y agois graph execution used for training only or inference also?
- LoganDark 3y agoInference. It's a big bottleneck for RWKV.cpp, second only to the matrix multiplies.
- make3 3y agodoes tbb work with apple Silicon?
- yukIttEft 3y agoI guess https://formulae.brew.sh/formula/tbb https://formulae.brew.sh/formula/tbb
- mhh__ 3y agoIt's not a very good library IMO.
- dindresto 3y agoggml or Intel TBB?
- mliker 3y agocongrats! I was just listening to your changelog interview from months ago in which you said you were going to move on from this after you brush up the code a bit, but it seems the momentum is too great. Glad to see you carrying this amazing project(s) forward!
- okhuman 3y agoThe establishment of ggml.ai a company focusing ggml and llama.cpp, the most innovative and exciting platform to come for local LLMs, on a Open Core model is just laziness. Just because you can (and have the connections), doesn't mean you should. It's a sad state of OSS when the best most brightest developers/founders reach for antiquated models. Maybe we take up a new rules in OSS communities that say you must release your CORE software as MIT at the same time you plan to go Open Core (and no sooner). Why should OSS communities take on your product market fit?!
- wmf 3y agoThis looks off-topic since GGML has not announced anything about open core and their software is already MIT. More generally, if you want to take away somebody's business model you need to provide one that works. It isn't easy.
- okhuman 3y agoAgreed with you 100% - its not easy. Sometimes I just wish someone as talented as Georgi would innovate not just on the core tech side but bring that same tenancy to the licensing side, in a way that aligns incentives better and tries out something new. And that the community would have his back if some new approach failed, no matter what.
- conjecTech 3y agoCongratulations! How do you plan to make money?
- ggerganov 3y agoI'm planning to write code and have fun!
- jgrahamc 3y agoThis is a good plan.
- az226 3y agoHave you thought about what your path looks like to get to the next phase? Are you taking on any more investors pre-seee?
- beardog 3y ago>ggml.ai is a company founded by Georgi Gerganov to support the development of ggml. Nat Friedman and Daniel Gross provided the pre-seed funding. Did you give them a different answer? It is okay if you can't or don't want to share, but I doubt the company is only planning to have fun. Regardless, best of luck to you and thank you for your efforts so far.
- aryamaan 3y agoCould someone at high level talk more about how one starts contributing to this kind of problems. For the people who build solutions for data handling— ranging from crud to building highly scalable solutions— these things are alien concepts. (Or maybe I am just talking about it myself)
- doxeddaily 3y agoThis scratches my itch for no dependencies.
- pawelduda 3y agoI happen to have RPi 4B with HomeAssistant. Is this something I could set up on it and integrate with HA to control it with speech, or is it overkill?
- boppo1 3y agoI doubt it. I'm running 4-bit 30B and 65B models with 64GB ram, a 4080 and a 7900x. The 7B models are less demanding, but even so, You'll need more than an rpi. Even then, it would be a project to get these to control something. This is more 'first baby steps' toward the edge.
- pawelduda 3y agoThe article shows example running on RPI that recognizes colour names. I could just come up with keywords that would invoke certain commands and feed them to HA, which would match them to an automation (i.e. turn off kitchen, or just kitchen ) . I think a PoC is doable, but I'm aware I could run into limitations quickly. Idk might give it a try when I'm bored. Would love voice assistant running locally but probably there are solutions out there - didn't get to do the research yet
- kkielhofner 3y agoShameless plug, I'm the founder of Willow[0]. In short you can: 1) Run a local Willow Inference Server[1]. Supports CPU or CUDA, just about the fastest implementation of Whisper out there for "real time" speech. 2) Run local command detection on device. We pull your Home Assistant entities on setup and define basic grammar for them but any English commands (up to 400) are supported. They are recognized directly on the $50 ESP BOX device and sent to Home Assistant (or openHAB, or a REST endpoint, etc) for processing. Whether WIS or local our performance target is 500ms from end of speech to command executed. [0] - https://github.com/toverainc/willow https://github.com/toverainc/willow [1] - https://github.com/toverainc/willow-inference-server https://github.com/toverainc/willow-inference-server
- 3y ago
- huevosabio 3y agoVery exciting! Now, we just need a post that benchmarks the different options (ggml, tvm, AItemplate, hippoml) and helps deciding which route to take.
- Dwedit 3y agoThere was a big stink one time when the file format changed, causing older model files to become unusable on newer versions of llama.cpp.
- edfletcher_t137 3y agoThis is a bang-up idea, you absolutely love to see capital investment on this type of open, commodity-hardware-focused foundational technology. Rock on GGMLers & thank you!
- graycat 3y agoWOW! They are using BFGS! Haven't heard of that in decades! Had to think a little: Yup, the full name is Broyden–Fletcher–Goldfarb–Shanno for iterative unconstrained non-linear optimization! Some of the earlier descriptions of the optimization being used in the AI learning was about steepest descent, that is, just find the gradient of the function are trying to minimize and move some distance in that direction. Just using the gradient was concerning since that method tends to zig zag where after, say, 100 iterations the distance moved in the 100 iterations might be several times farther than the distance from the starting point of the iterations to the final one. Can visualize this zig zag already in just two dimensions, say, following a river, say, a river that curves, down a valley the river cut over a million years or so, that is, a valley with steep sides. Then gradient descent may keep crossing the river and go maybe 10 feet for each foot downstream! Right, if just trying to go downhill on a tilted flat plane, then the gradient will point in the steepest descent on the plane and gradient descent will go all way downhill in just one iteration. In even moderately challenging problems, BFGS can a big improvement.
- KronisLV 3y agoJust today, I finished a blog post (also my latest submission, felt like could be useful to some) about how to get something like this working in a bundle of something to run models, as well as a web UI for more easy interaction - in my case that was koboldcpp, which can run GGML, both on the CPU (with OpenBLAS) and on the GPU (with CLBlast). Thanks to Hugging Face, getting Metharme, WizardLM or other models is also extremely easy, and the 4-bit quantized ones provide decent performance even on commodity hardware! I tested it out both locally (6c/12t CPU) and on a Hetzner CPX41 instance (8 AMD cores, 16 GB of RAM, no GPU), the latter of which costs about 25 EUR per month and still can generate decent responses in less than half a minute, my local machine needing approx. double that time. While not quite as good as one might expect (decent response times mean maxing out CPU for the single request, if you don't have a compatible GPU with enough VRAM), the technology is definitely at a point where it's possible for it to make people's lives easier in select use cases with some supervision (e.g. customer support). What an interesting time to be alive, I wonder where we'll be in a decade.
- SparkyMcUnicorn 3y agoSeems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/workers/gpt-neox https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq https://modal.com/docs/guide/ex/falcon_gptq
- sva_ 3y agoReally impressive work and I've asked this before, but is it really a good thing to have basically the whole library in a single 16k line file?
- regularfry 3y agoIt makes syncing between llama.cpp, whisper.cpp, and ggml itself quite straightforward. I think the lesson here is that this setup has enabled some very high-speed project evolution or, at least, not got in its way. If that is surprising and you were expecting downsides, a) why; and b) where did they go?
- mhh__ 3y agohttps://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules
- regularfry 3y agoOr... `cp`. It's fine.
- CamperBob2 3y agoYes. Next question
- baobabKoodaa 3y agoI guess the "clean code" crowd would like to refactor this into hundreds of files that all call each other in an incomprehensible maze, plus pulling in 20GB of dependencies from the internet during install. Because that is the way™.
- boringuser2 3y agoLooking at the source of this kind of underlines the difference between machine learning scientist types and actual computer scientists.
- baobabKoodaa 3y agoCan you elaborate?
- boringuser2 3y agohttps://github.com/ggerganov/ggml/blob/master/src/ggml-opencl.c https://github.com/ggerganov/ggml/blob/master/src/ggml-openc...
- baobabKoodaa 3y agoWow, what a passive aggressive response. So, I take it that you can't elaborate. Got it.
- iamflimflam1 3y agoI've always thought on the edge to be IoT type stuff. So running on embedded devices. But maybe that not the case?
- Y_Y 3y agoLike any new term the (mis)usage broadens the meaning over time until it either it's widely known, it's unfashionable, or most likely; it becomes so broad as to be meaningless and hence it achieves buzzword apotheosis. My old job title had "edge" in it, and I still don't know what it's supposed to mean, although "not cloud" is a good approximation.
- b33j0r 3y agoSounds like your job had a lot of velocity with lateral tragmorphicity in Q1, just in time for staff engineer optimization! Nicely done. Here is ~$50 worth of stock.
- timerol 3y ago"Edge computing" is a pretty vague term, and can encompass anything from a 8MHz ARM core that can barely talk compliant BLE, all the way to a multi-thousand dollar setup on something like a self-checkout machine, which may have more compute available than your average laptop. In that range are home assistants, which normally have some basic ML for wake word detection, and then send the next bit of audio to the cloud with a more advanced model for full speech-to-text (and response)
- anentropic 3y agoHere it means more 'on your own device' rather than 'in the cloud' You could consider that the real edge, whereas edge computing often means 'at the edge of the cloud' i.e. local CDN node
- java_beyb 3y agoedge brings compute close to where data is generated, cloud brings data to compute. even processing something in a web browser is called edge. i guess due to this impression the industry is moving towards "on-device"
- s1k3s 3y agoI'm out of the loop on this entire thing so call me an idiot if I get it wrong. Isn't this whole movement based on a model leak from Meta? Aren't licenses involved that prevent it from going commercial?
- dimfeld 3y agoOnly the weights themselves. There have been other models since then built on the same Llama architecture, but trained from scratch so they're safe for commercial user. The GGML code and related projects (llama.cpp and so on) also support some other model types now such as Mosaic's MPT series.
- detrites 3y agoGGML is essentially a library of lego pieces that can be put together to work with many LLM or other types of ML models. Meta's leaked model is one for which GGML has been applied to for fast, local inference.
- ac29 3y agoIt wasn't a leak, LLaMa was released publicly under an open-ish license (the code is GPL, the model weights require registration and prohibit commercial use).
- zkmlintern 3y agoZKML fixes this
- FailMore 3y agoRemember
- FailMore 3y agoCommenting to remember. Looks good
- ex3ndr 3y agoso sad we still don't have a perfect neural network for activation word to make home assistants complete
- version_five 3y agoSo I'll say it: I understand why someone would do it, I'm sure they backed up a money truck, but it sucks to see this sell out. VC is going to suck all the value out and leave something that exists to funnel money to them. This has been an awesome project. I hope somebody forks it and maintains a version that isn't profit motivated.
- okhuman 3y agoI share the same sentiment and feel the same, its not you alone saying these things. It's starting to seem the llama.cpp project wasn't so community oriented to begin with - which in itself takes a lot of patience.
- swyx 3y ago> It's starting to seem the llama.cpp project wasn't so community oriented to begin with what do you mean? llama.cpp has 181 contributors.
- fallous 3y agoI don't mind a project that is profit motivated since it incentivizes the people working on the project as well as drives them to create solutions that solve customer problems. I do, however, have a problem with VC backing due to their frequent insistence on pushing for short-term maximal share pricing rather than long-term value. Modern VC also seem to suffer from a fatal mixture of hubris and narcissism, certain of their genius and business acumen despite all evidence to the contrary. When advising start-ups who are taking on VC funding and the inevitable "business guidance" they push on founders I like to remind them that even the best investors are only good at investing. If they were good at business, they wouldn't bother with investing in other companies but instead start companies themselves since they would own 100% and maximize their returns.
- Tan-Aki 3y agoCan anyone explain to me, in simple terms, and at a high level, what the heck am I looking at? What is this library for? What does it mean "it is used by lama.cpp and whisper.cpp"? How is it revolutionary? Thank you very much in advance!
- Tan-Aki 3y agoWith maybe a tiny bit of history as well? Pretty please? : p
- orost 3y agoggml is a library that provides operations for running machine learning models llama.cpp is a project that uses ggml to run LLaMA, a large language model (like GPT) by Meta whisper.cpp is a project that uses ggml to run Whisper, a speech recognition model by OpenAI ggml's distinguishing feature is efficient operation on CPU. Traditionally, this sort of work is done on GPU, but GPUs with large amounts of memory are specialized and extremely expensive hardware. ggml achieves acceptable speed on commodity hardware.
- Tan-Aki 3y agoThank you so much for your kindness Orost. Sharing really IS caring. I understand. May good things happen to you. Peace.
- Tan-Aki 3y agoThank you so much for your kindness Orost. Sharing really IS caring. I understand. May good things happen to you. Peace.
- noman-land 3y agoGeorgi if you're reading this, I've had a lot of fun with whisper.cpp llama.cpp because of you so thank you very much.
- hanselot 3y agoI envy his drive and ambition. I can't force myself to finish writing a simple alarm clock app for android, never mind pathing the literal road to the future of Open Source AI. Would someone else have taken his place had he not been around? Maybe, but I'm insanely happy that he is around. The amount of hours I've sunk into LLM's is crazy, and it's mostly thanks to his work that I can both run and download models in meaningful timeframes. And yes, I have tested llama.cpp on my android and it works 100% on termux. (Your biggest enemy here will be Android process reaper when you hit the memory cap)
- ffvvtvtbyh 3y agoKeen to work together. I also struggle with follow through
- legendofbrando 3y agoRunning whisper locally on my iPhone back in December and watching perfect transcriptions pop out without sending anything to a server was a real lightbulb moment for me that set in motion a bunch of the work I’m doing now. Excited to see the new heights this unlocks!
- bugglebeetle 3y agoWhat are you using to run Whisper locally?
- ariym 3y agowhisper.cpp is optimized for Apple Silicon and is available as a Swift package https://github.com/ggerganov/whisper.spm https://github.com/ggerganov/whisper.spm
- kyt 3y agoI used the GGML version of Whisper and I had to revert back to the PyTorch version released by OpenAI. The GGML version simply didn't work well even for the same model. I am assuming it has to do with the quantization.
- ankitg12 3y agoQuite impressive, able to run a LLM on my local mac % ./bin/gpt-2 -m models/gpt-2-117M/ggml-model.bin -p "Let's talk about Machine Learning now" main: seed = 1686112244 gpt2_model_load: loading model from 'models/gpt-2-117M/ggml-model.bin' gpt2_model_load: n_vocab = 50257 gpt2_model_load: n_ctx = 1024 gpt2_model_load: n_embd = 768 gpt2_model_load: n_head = 12 gpt2_model_load: n_layer = 12 gpt2_model_load: ftype = 1 gpt2_model_load: qntvr = 0 gpt2_model_load: ggml tensor size = 224 bytes gpt2_model_load: ggml ctx size = 384.77 MB gpt2_model_load: memory size = 72.00 MB, n_mem = 12288 gpt2_model_load: model size = 239.08 MB extract_tests_from_file : No test file found. test_gpt_tokenizer : 0 tests failed out of 0 tests. main: prompt: 'Let's talk about Machine Learning now' main: number of tokens in prompt = 7, first 8 tokens: 5756 338 1561 546 10850 18252 783 Let's talk about Machine Learning now. The first step is to get a good understanding of what machine learning is. This is where things get messy. What do you think is the most difficult aspect of machine learning? Machine learning is the process of transforming data into an understanding of its contents and its operations. For example, in the following diagram, you can see that we use a machine learning approach to model an object. The object is a piece of a puzzle with many different components and some of the problems it solves will be difficult to solve for humans. What do you think of machine learning as? Machine learning is one of the most important, because it can help us understand how our data are structured. You can understand the structure of the data as the object is represented in its representation. What about data structures? How do you find out where a data structure or a structure is located in your data? In a lot of fields, you can think of structures as main: mem per token = 2008284 bytes main: load time = 366.33 ms main: sample time = 39.59 ms main: predict time = 3448.31 ms / 16.74 ms per token main: total time = 3894.15 ms
- dangrover 3y agoThere's so much potential for this as a tech powering all sorts of products, I hope it doesn't just become some Tarsnap type thing. (judging from initial site)
- cperciva 3y agoWhat's wrong with "Tarsnap type things"?
- jeadie 3y agoI'm very glad that this has some added funding. I am building a serverless API on the cloudflare edge network using GGML as the backbone --> tryinfima.com