menaerus

38 ms·

811.
by menaerus 2y ago
Now remind us what HW did we need to run local inference of llama2-69B (July, 2023)? And then contrast it to the HW we need to run llama3.1-70B (July, 2024)? In particular, which optimizations and in what way did they dramatically cut down
812.
by menaerus 2y ago
I think that the work looks quite interesting but it seriously lacks some important points to be covered. Benchmarks [1] only cover the random insert workload. Why doesn't it include other types of workloads? Inserting into the hashmap
813.
by menaerus 2y ago
Right. Have a look at the paper above from Meta on how they fine-tuned the Code Llama with LLVM IR to beat the compiler in producing size-optimized binaries.
814.
by menaerus 2y ago
I occasionally read their blogs, and I haven't looked into the source for a while, but application developers having to deal with the OOM because one is using custom memory allocation policy that doesn't deal with or suffers from
815.
by menaerus 2y ago
Makes no sense. Source?
816.
by menaerus 2y ago
> That's exactly what everybody advised me against doing - finetuning on own projects Why would someone advise against it? IMHO that sounds as the end game to me. If it weren't so darn expensive, I'd try this for myself fo
817.
by menaerus 2y ago
Sorry, I think that's delusional and if anything maybe only applicable to the very narrow and simple problem space. Nobody writes out the "spec" for the next best inference engine or the next best distributed database.
818.
by menaerus 2y ago
Senior who doesn't deliver (write code) is not a senior. Beating about the bush in meetings is easy, executing the stuff isn't.
819.
by menaerus 2y ago
> It’s C++ 20/23 without virtual memory Not sure what you mean by this? Virtual memory is implied by the CPU MMU and consequently OS kernel. Perhaps you meant they use a lot of custom memory allocation schemes? Otherwise, I agree th
820.
by menaerus 2y ago
Correct, you need to build an AST representation of the code that you want to index. Essentially, it's a compiler frontend pass and which is why it takes so much longer than what ctags heuristics do. Now think millions of lines of code
821.
by menaerus 2y ago
And now also consider the amount of time, school, university, practice and $$$ it took to train a software engineer. Repeat for the population of presumably ~30 million software engineers in the world. Time-wise it doesn't scale at all
822.
by menaerus 2y ago
Splitting the code into 10s or 100s repositories is a cancer. It's almost as if you're intentionally trying to make your dev life miserable by pretending that you speeded up the dev turnaround times by not running the full test ci
823.
by menaerus 2y ago
We absolutely care about absolute costs. 70B model this year will cost as much as it will next year, unless Nvidia decides to lose their profits. The question is whether an inference cost is dropping down. And the answer is obviously no. I
824.
by menaerus 2y ago
They show a test-run of a 1B llama-3.2 model. Doesn't that fit in a single mac? Distributing the workload in this case must be slower than running it on a single machine. However, this is interesting and I'm confused why aren&#x27
825.
by menaerus 2y ago
Sure, let's say that 8B llama3.1 gets comparable performance of it's 70B llama2 predecessor. Not quite true but let's say that hypothetically it is. That still leave us with 70B llama3.1. How much VRAM and inference compute i
826.
by menaerus 2y ago
In August 2023, llama2 34B was released and at that time, without employing model quantization, in order to fit this model one needed to have a GPU, or set of GPUs, with total of ~34x2.5=85G of VRAM. That said, can you be more specific what
827.
by menaerus 2y ago
I agree that they can be powerful but I guess, as always, it doesn't apply in the general case. For example, AFAIK hw prefetcher runs at 32B or 64B per cycle which basically means that you're only going to see a benefit from it if
828.
by menaerus 2y ago
Those are the (subsidized) prices that end clients are paying for the service so that's not something that is representative of what the actual inference costs are. Somebody still needs to pay that (actual) price in the end. For infere
829.
by menaerus 2y ago
No. For 98% of the multi-core sensitive code I don't have nor I need shared ownership. While C++ doesn't force you into such semantics but provides you with the ability to do so, the Rust semantics and compiler pesimisstically do.
830.
by menaerus 2y ago
The stakes of deploying a buggy code into the data center production code can easily cost millions of $$$ and yet we still see that one of the primary usages of LLMs today is exactly in the software engineering. Accountability exists in eve
831.
by menaerus 2y ago
Easier to be said than done since that's one of the core tradeoffs of the Rust language. Language is forcing these semantics upon you and while it is possible to get around it with unsafe blocks it turns out to be much more difficult i
832.
by menaerus 2y ago
And then I guess you're also missing the point that interpreting and writing the code also requires an expert and that in that respect it is no different than law. I could argue that engineering is more complex than interpreting law bu
833.
by menaerus 2y ago
The point is not in replacing all of the lawyers or programmers but rather that we will no longer need so many of them since a lot of their expertise is becoming a commodity today. This is a fact and there have been many many examples of th
834.
by menaerus 2y ago
So, fine-tuning the model to the law of the exact country. Or fine-tuning the model to the problem space of the exact codebase/product. You hire 10 law experts instead of 100. Or you hire 10 programmers instead of 100. Expertise is bec
835.
by menaerus 2y ago
In theory none but in practice the codebase ends up littered with hidden shared state mostly disguised through one or another shared pointer implementation. And this happens because that's what the Rust compiler is pessimistically enfo
836.
by menaerus 2y ago
Modules? Yeah, I don't expect them to be a game changer either. Exactly because of the reasons I mentioned.
837.
by menaerus 2y ago
> unless IBM specifically fine-tuned an AI for those languages based on their internal documentation and code. Why do you think this isn't already or won't be a case in the near future? Because that's exactly what I believ
838.
by menaerus 2y ago
> code is a much smaller domain than the law. Law of an average EU country fits in several hundred, let's say even thousands, of pages of text. Specification. Very well known. Low frequency of updates. But code? Everything opposite
839.
by menaerus 2y ago
Counter-argument: code does not exist in vacuum yet we have small and mid-sized LLMs that can already output reasonable code.
840.
by menaerus 2y ago
I don't understand the limitation, e.g. how much data do you need to train the "law state" specific LLM that doesn't know anything else but that? Such LLM does not need to have 400B of parameters since it's not a ge

More ›