lhl

28 ms·

271.
by lhl 3y ago
A few years ago AMD split off their GPU architectures to CDNA (focused on data center compute) and RDNA (focused on rendering for gaming and workstations). This in itself is fine and what Nvidia was already doing, it makes sense to optimize
272.
by lhl 3y ago
In the data center, I think AMD is a lot more viable than most people think. MosaicML recently did a test and were able to swap MI250s with A100s basically seamlessly, within a single training run even, and ran into no issues: https:/
273.
by lhl 3y ago
As an article aimed towards newbies I think it's pretty good, but I'd agree "comprehensive" is a bit of a reach. I think a better title would have been something like "Get LLama2 running locally on every platform ea
274.
by lhl 3y ago
The benefits are primarily price - 96GB of VRAM would be 4x3090/4090 (~$6K) or 2xA6000 (~$8-14K) cards (also, looks like you can buy an 80GB A100 PCIe for about $15K atm). While Apple is using LPDDR5, it is also running a lot more chan
275.
by lhl 3y ago
Just a quick note, it's worth pointing out that for most people (eg, wanting to chat to a model in realtime), I don't think running locally on a CPU is a very viable option unless you're very patient. On my 16c Ryzen 5950X&
276.
by lhl 3y ago
As a followup on LLama2's prompting, here's a good thread with some more details: https://www.reddit.com/r/LocalLLaMA/comments/155po2p/get_lla... (see also: https://www.reddit.com&#x
277.
by lhl 3y ago
While on the surface, a 192GB Mac Studio seems like a great deal (it's not much more than a 48GB A6000!), there are several reasons why this might not be a good idea: * I assume most people have never used llama.cpp Metal w/ large
278.
by lhl 3y ago
Just as a followup for those interested, after reading the paper and some more critique: The paper is mostly ok in that it points out model drift and the importance of making sure you use API versions. One good thing is that the full datase
279.
by lhl 3y ago
old user here, just want to confirm that this tool is super neat, can see it being very useful for comparing project health when picking a lib to use. hope it can be sustainable/stick around in the future.
280.
by lhl 3y ago
I was using the GPT-4 base model for pairing when it launched, I'm sure it's still fine, but Code Interpreter I think is just a nice upgrade since it has a very full-featured VM and Python w/ several hundred libs built in, it
281.
by lhl 3y ago
Hmm, I tested twice with GPT-4 and it got the answer wrong both times but it did think step by step... The first time it gets to "17077 divided by 13 is exactly 1313, a whole number." [1] and the second time it concludes "H
282.
by lhl 3y ago
I have not read the paper yet (in my backlog, here's the paper: https://arxiv.org/pdf/2307.09009.pdf ), but it's important note that the paper is entitled "How Is ChatGPT’s Behavior Changing over Time?&qu
283.
by lhl 3y ago
ExLlama still uses a bit less VRAM than anything else out there: https://github.com/turboderp/exllama#new-implementation - this is sometimes significant since from my personal experience it can support full context on
284.
by lhl 3y ago
I'd suggest checking out ServeTheHome, which has been reviewing tons of mini PCs or CNX Software or Liliputing which cover all the SBCs and minipcs that get announced/launched. There are way too many options to list and it largely
285.
by lhl 3y ago
For those getting started, the easiest one click installer I've used is Nomic.ai's gpt4all: https://gpt4all.io/ This runs with a simple GUI on Windows/Mac/Linux, leverages a fork of llama.cpp on the back
286.
by lhl 3y ago
I'd argue that if the system cannot be trained output the results of system calls faithfully (this was not an adversarial test), then it simply has been badly trained, but even ignoring that, as you mention, system call output executio
287.
by lhl 3y ago
Until recently, exllama was significantly faster, but they're about on par now (with llama.cpp pulling ahead on certain hardware or with certain compile-time optimizations now even). There are a couple big difference as I see it. llama
288.
by lhl 3y ago
Ha, yeah, I was going to say... the writing was sadly pretty much on the wall for a while it seemed like. SimplyNUC (led by ex-Intel NUC guys) will still be chugging on, but for enterprise/business use, personally I actually recommend
289.
by lhl 3y ago
I think you're making some improper assumptions about my familiarity with the technology. The whole point of adding grounding via external plugins, code interpreters, or retrieval augmentation is to reduce hallucinations - if a block o
290.
by lhl 3y ago
It's one thing to say to "use code with caution" but it's another thing to pretend to run a calculation and then hallucinate the answer (or hallucinate that it's running code). I just tried out this exact example. M
291.
by lhl 3y ago
ChatGPT isn't exactly aware of what packages it has available. If it says it can't, you can just ask it to nicely to try. Here's a list of what it has installed currently: https://github.com/petergpt/code
292.
by lhl 3y ago
Why would you only update the original base model and not simply continue fine-tuning? (This is a rhetorical question of course, and there are multiple empirical ways to confirm the majority of your answer is "hallucinated," not l
293.
by lhl 3y ago
Well there's OpenAI's release notes for one: https://help.openai.com/en/articles/6825453-chatgpt-release-... Pre-training of a foundational model is what you're thinking of for the "absurdly ex
294.
by lhl 3y ago
So, I went down an interesting rabbit hole with this. Claude was willing to give me (correctly) 100 digits of Pi, but when asked to write some Python code to let me confirm the digits of Pi, it failed over and over again (in different ways
295.
by lhl 3y ago
While that's what the Technical Report ( https://arxiv.org/pdf/2303.08774v3.pdf ) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably o
296.
by lhl 3y ago
Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). (Found in model card: pass@1) For comparison: * GPT-4 claims 85.4 on HumanEval, in a recent paper ht
297.
by lhl 3y ago
Most of these MiniPC companies like Beelink, Minisforum, Morefine, etc are small Shenzen-based operations so personally I wouldn't expect much in terms of support (whether it's BIOS updates or warranty). If you're going to ge
298.
by lhl 3y ago
A few years ago I did a bunch of poking w/ a 4800H, you can see the actual mechanics of using RyzenAdj. There are also Geekbench 5 benchmarks linked from the doc at different power limits which you can compare to 4800U devices on Geekb
299.
by lhl 3y ago
For those interested in seeing more about how these new 7040 APUs (Zen4 + RDNA3 "Phoenix") perform, Notebookcheck just reviewed a 7840HS (65W max TDP I believe) minipc: https://www.notebookcheck.net/AMD-Ryzen-7-784
300.
by lhl 3y ago
I agree that everyone should do their own cost-benefit analysis, especially if they have to buy additional hardware (used RTX 3090s are ~$700 atm), but one important thing to note for those running the numbers is that all your tokens need t

More ›