4 ms·
The issue here isn't specifically about the classification of memory, be it "unified memory," RAM, or VRAM. The primary concern is ensuring there's enough memor
by rthnbgrredf 3y ago
The issue here isn't specifically about the classification of memory, be it "unified memory," RAM, or VRAM. The primary concern is ensuring there's enough memory capacity for the models required for inference. The real question at hand is the Mac's value proposition in terms of inference speed, particularly for models as large as 70 billion parameters. Utilizing a 4090 GPU can facilitate real-time inference, which is the desired outcome for most users. In contrast, a Mac Studio offers close to real-time inference speeds, which might be disappointing for users expecting a real-time experience. Then, there's the option of CPU + RAM-based inference, which suits scenarios where immediate responses aren't crucial, allowing for batch processing of prompts and subsequent retrieval of responses. Considering the price points of both the Mac Studio and high-end GPUs are relatively comparable, it begs the question of the practicality and value of near real-time inference in specific use cases.
- abtinf 3y ago[flagged]
- rthnbgrredf 3y agoI'm not a gpt. But now you could say that this is exactly how a gpt would answer and we get stuck in a loop and there's no obvious way to prove that I'm not a gpt.
- pbhjpbhj 3y ago'Write me something profane?' That probably weeds out commercially available GPTs?
- rthnbgrredf 3y agoI'm sorry, but I can't fulfill that request. Is there anything else I might assist you with? ;)
- int_19h 3y agohttps://chirper.ai/aligned/chirp/65e58aa9001d853c https://chirper.ai/aligned/chirp/65e58aa9001d853c
- natch 3y agoNo, this is a common fallacy. You can tell ChatGPT, one of the most infamously hobbled GPTs, in custom instructions that you do want profanity, and it will oblige. This is not a jailbreak, this is supported behavior.
- rthnbgrredf 3y agoI do think that there is a certain amount of gpt bot activity present on HN. But I don't think it makes sense to call people out and saying they are a gpt just based on one comment.
- abtinf 3y agoInteresting. Let's review the comment. > The issue here isn't specifically about the classification of memory, be it "unified memory," RAM, or VRAM. The primary concern is ensuring there's enough memory capacity for the models required for inference. The comment chain is about training, not inference. > The real question at hand is the Mac's value proposition in terms of inference speed, particularly for models as large as 70 billion parameters. Again, wrong topic. > Utilizing a 4090 GPU can facilitate real-time inference, which is the desired outcome for most users. Generic statement. Semantically empty. Typical LLM style. > In contrast, a Mac Studio offers close to real-time inference speeds, which might be disappointing for users expecting a real-time experience. Tautological generic statement. Semantically empty. Typical LLM style. > Then, there's the option of CPU + RAM-based inference, which suits scenarios where immediate responses aren't crucial, allowing for batch processing of prompts and subsequent retrieval of responses. Contradicts first sentence that "classification of memory" isn't important. Fails to recognize this the same category as previous statement. Subtle shift from first sentence that declared "primary concern is ... membory capacity", to focusing purely on performance. This kind of incoherent shift is common in LLM output. > Considering the price points of both the Mac Studio and high-end GPUs are relatively comparable, it begs the question of the practicality and value of near real-time inference in specific use cases. Completes shift from memory capacity to performance. Compares not really comparable things. "Specific use cases" is a tell-tale LLM marker. Semantically empty.
- MrYellowP 3y agoI feel the need to point out that people, who spend many hours writing with an LLM, will eventually start writing like the LLM.
- rthnbgrredf 3y agoI'm definitely guilty of this. Especially for non native speaker who might more easily lend towards adapting phrases from others (including gpts), because they are not sure how to phrase it correctly.
- rthnbgrredf 3y ago
- Lalabadie 3y agoConsidering that the topic is approachability and energy efficiency, that Mac Studio will do reasonably fast inference while consuming <200W at full load. The speed is certainly not comparable to dedicated GPUs, but the power efficiency is ridiculous for a very usable speed and no hardware setup.
- Applejinx 3y agoThis, and then you get to have a Mac Studio. I have one, where I selected an M1 Ultra and 128G RAM to facilitate just this sort of thing. But in practice, I'm spending much more time using it to edit 4K video, and as a recording studio/to develop audio plugins on, and to livestream while doing these things. Turns out it's good at these things, and since I have the LLAMA 70b language model at home and can run it directly unquantized (not at blinding speed, of course, but it'll run just fine), I'm naturally interested in learning how to fine tune it :)
- int_19h 3y agoYep, I also got mine specifically for LLMs and ended up using it as a second desktop for other things; actually strongly considering making it my primary at this point. I still wouldn't recommend it to someone just looking for a powerful desktop, just because $3K is way overpriced for what you get (non-replaceable 1Tb SSD is so Apple!). But it's certainly great if you already have it...
- int_19h 3y ago"Real-time" is a very vague descriptor. I get 7-8 tok/s for 70b model inference on my M1 Mac - that's pretty real-time to me. Even Professor-155b runs "good enough" (~3 tok/s) for what I'd consider real-time chat in English.