brucethemoose2

8 ms·

31.
by brucethemoose2 3y ago
My concern is that Elon has shut the door with a sloppy lawsuit. If some other entity (Government agency? Another company?) Comes around to pull this string, its going to set a bad precedent.
32.
by brucethemoose2 3y ago
- A: I didn't know this, awesome. - B: Yes, uploading via a web page would be nice too. I have seen other apps do this.
33.
by brucethemoose2 3y ago
Is there an equivalent of this with an http front end, so the client doesn't need to install anything? I have used such apps before, but it seems they have gone unmaintained and don't work anymore.
34.
by brucethemoose2 3y ago
Yi 9B has been released with quite impressive looking benchmarks. While probably not a SOTA coding model, it looks like a strong competitor to leading edge "small" models Mistral 0.2 and Solar. Meanwhile, Yi-34B-200K has been upda
35.
Kai-Fu Lee's 01.ai Releases 9B, Long Context 34B Models(huggingface.co)
2 points by brucethemoose2 3y ago|2 comments
36.
by brucethemoose2 3y ago
Apple could dramatically improve performance if they just tasked one Metal engineer on llama.cpp. Like, just to finish up flash attention and quantum KV cache, and optimize the Metal kernels. I wouldn't be surprised if they could dou
37.
by brucethemoose2 3y ago
AI is unique in that the incentive is to write engaging, compelling, perhaps wonderful ostensibly truthful content that is total nonsense . Enthusiasts in a subject are often wrong, but usually the goal is at least to convey something us
38.
by brucethemoose2 3y ago
On the contrary, I think open human discourse in platforms that aren't really a lure for spammers or trolls (like code repos or really niche topics) is more important than ever. Generic social media is going to be a hellscape, yes, but
39.
by brucethemoose2 3y ago
One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes. Running one as a
40.
by brucethemoose2 3y ago
I passed over the 7900 XTX because its not worth the trouble over a 3090. That calculus totally changes with 48GB. I would put up with A LOT of trouble for that.
41.
by brucethemoose2 3y ago
Its insane that a 2020 gaming GPU is such a niche for deep learning. Thats ancient history in the world of graphics, and its only that way because the market is straight up anticompetitive (and AMD is too dense to break out of it with a 4
42.
by brucethemoose2 3y ago
How about higher capacity GDDR6X? 48GB consumer cards (or 96GB pro cards) would sell like hotcakes if AMD/Intel dare to break the artificial VRAM segmentation status quo.
43.
by brucethemoose2 3y ago
Asus uses custom firmware for more performance consistency, but yeah the silicon is the same AFAIK.
44.
by brucethemoose2 3y ago
I would not trust AMD CPUs either. All modern desktop CPUs on most motherboards, especially those at the high end, will clock themselves to ridiculous voltages/power levels just to squeeze out another 5% performance. Intel is particul
45.
by brucethemoose2 3y ago
Nah. Reddit will escape the enshittification cycle, just like... Uh...
46.
by brucethemoose2 3y ago
In hindsight I think the post was overly negative, even though I am no Gemma fan. People are downvoting just because of that. Now, what's a little odd to me is that there was so much interest positive interest from HNers? People submit
47.
by brucethemoose2 3y ago
exl2 is Nvidia/AMD only. But GGUF Mixtral should fit in 32GB... just not with the full 32K context. Long context is very memory intense in llama.cpp, at least until they fully implement flash attention and a quantized cache.
48.
by brucethemoose2 3y ago
On 33B/34B models I get 35 tokens/sec, way faster than I can read streaming in. At huge contexts (like 30K-74K), prompt processing takes forever and token generation is slower, but its still faster than I can read. Miqu 70B is slo
49.
by brucethemoose2 3y ago
Yi 34 200K finetunes (like Tess 1.5),Deepseek Code 33B and Miqu 70B definitely outpace ChatGPT-3.5, at least for me. They don't have the augmentations of being a service, but generally they are smarter, have a bigger context and (perha
50.
by brucethemoose2 3y ago
Counterpoints: - Local models are pretty easy to de-censor, if thats what you mean. - ...Yeah, it should not be labeled as a 7B. Its sort of 7B class. - The repo mentions they use the llama-cpp-python server - 1M context brute forced across
51.
by brucethemoose2 3y ago
Practically, speaking, I OOM'd running Gemma on a 3090 using a config that had VRAM to spare for Mistral 7B. It kinda surprised me at first, until I realized why.
52.
by brucethemoose2 3y ago
What are HNers looking for in this article? The architectural differences, or how to run/finetune it?
53.
by brucethemoose2 3y ago
I just loaded it in vllm with default settings. I can't share the eval, but it's pretty simple: it asks a question about some data, and is restricted to only answer yes/no (based on the output logits and suggested in the prom
54.
by brucethemoose2 3y ago
The scale of the cloning is quite fascinating. As I said, the entire world needs to shift how they percieve digital media. People are used to seeing a photo/video and thinking of it as representing something real... that's not the
55.
by brucethemoose2 3y ago
OK that sounds lovely. We are but a poor startup, but I am hoping we can afford to reach out to y'all soon. Maybe even share workflows/stuff that works for us. If some ML (unsloth? lorax? sglang?) work on a Mi300 with a little t
56.
by brucethemoose2 3y ago
We can't quite afford an 8x bare metal instance, but I am pinning this for later. Think ya'll will ever offer single MI300X instances?
57.
by brucethemoose2 3y ago
Some of these are pretty good. The AITemplate Stable diffusion demo, for instance, is a nice hidden gem (though I'm not sure if it works on consumer hardware these days).
58.
by brucethemoose2 3y ago
It's complicated, but basically because most are llama architecture. Meta all but set the standard for open source llms when they released llama1, and anyone trying to deviate from it has run into trouble because the models don'
59.
by brucethemoose2 3y ago
I eval'd it with vllm. One thing I do suspect people are running into is sampling issues. Gemma probably doesn't like llama defaults with its 256K vocab. Many Chinese llms have a similar "default sampling" issue. But o
60.
by brucethemoose2 3y ago
At the risk of being off topic: I am jealous of your handle. And yes, this was an amazing landing and proof of the tech.

More ›