5 ms·

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if you squint this is a useful/convincing "ChatGPT on a laptop" minus the corporate lobotomy only a few short months after release. Very exciting! I actually care about the upcoming Mac Pro now. $3999 Mac Studio with 64GB ram. +$800 for 128GB.

londons_explore 4y ago
LLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"
- RoboTeddy 4y ago
  LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```
  - guestbest 4y ago
    This is Commodore 64 tier answers
    - circuit10 4y ago
      That's just an example
    - lgas 4y ago
      That's the point, even C64 tier examples in one shot or few shot learning do wonders for changing the behavior of the model.
- rastapasta42 4y ago
  Ask dumb questions receive dumb answers.
- nabakin 4y ago
  It performs very well, but you have to give it the right prompt and model params. I imagine it will be ChatGPT level once it is trained with RLHF
alphanullmeric 4y ago
Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.
- slaymaker1907 4y ago
  Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data. However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time). Really, we just need some good systems devs working on running these huge models.
- PragmaticPulp 4y ago
  PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.
  - recuter 4y ago
    That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
    - smoldesu 4y ago
      puts on my jaded realist hat It's a great option if you have the hardware and want the speed. It's table stakes when other vendors like Nvidia, Intel, Qualcomm and Microsoft have acceleration though. Raw-compute-for-the-buck has always been a blowout with Apple Silicon GPUs, and it's not any prettier now that 4nm 40XX series cards are available. Hell, an Intel A770 with 16gb of VRAM is still cheaper than adding 16gb of RAM to a Mac Mini. It's good stuff, but pitched a bit hard with all the marketing. From where I'm standing, it looks like Apple is playing catch-up with their GPUs and acceleration tech. tentatively removes hat of jaded realism
Nevermark 4y ago
Given that the RAM is directly accessible by CPU, GPU and DPU/Neural cores, it really is premium RAM. Apple's visionary hardware team has finally caught up with Apple's visionary high RAM prices! :)
- DennisAleynikov 4y ago
  Also um... Idk if anyone cares but apple didn't do anything to make unified memory. All arm chips do this lmao Apple just has the best known arm chips with the highest mobile performance (yes faster server arm chips exist too)
  - fooker 4y ago
    It’s amazing how Apple “doesn’t do” anything, but manages to define industry trends over and over again!
    - 7to2 4y ago
      That's what brand recognition and brand loyalty (and the money they generate) allows you to do. This is not necessarily a bad thing since it forces other competitors - who like to artificially limit their products - to actually get of their greedy asses and compete.
      - DennisAleynikov 4y ago
        Yeah which is why... Unified memory on Chromebooks and android devices as well as surface pro serve the same purpose. It's marketed as an upsell by apple because it's an advantage of ARM but many people will never care to learn what that means
deleted 4y ago
[deleted]

londons_explore 4y ago

LLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"

RoboTeddy 4y ago
LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```
- guestbest 4y ago
  This is Commodore 64 tier answers
  - circuit10 4y ago
    That's just an example
  - lgas 4y ago
    That's the point, even C64 tier examples in one shot or few shot learning do wonders for changing the behavior of the model.
rastapasta42 4y ago
Ask dumb questions receive dumb answers.
nabakin 4y ago
It performs very well, but you have to give it the right prompt and model params. I imagine it will be ChatGPT level once it is trained with RLHF

alphanullmeric 4y ago

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

slaymaker1907 4y ago
Nvidia actually does support this via unified memory. It’s actually an amazing performance trick since you can avoid launching a bunch of kernels just to ferry data back and forth. I did this on a a GTX 1080 for a school project to speed up random shuffling of large amounts of data. However, even without this feature, you can implement this sort of thing manually in most cases and you’re already being careful on a GPU to respect the cache (only working with one contiguous set of data of memory at a time). Really, we just need some good systems devs working on running these huge models.
PragmaticPulp 4y ago
PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.
- recuter 4y ago
  That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
  - smoldesu 4y ago
    puts on my jaded realist hat It's a great option if you have the hardware and want the speed. It's table stakes when other vendors like Nvidia, Intel, Qualcomm and Microsoft have acceleration though. Raw-compute-for-the-buck has always been a blowout with Apple Silicon GPUs, and it's not any prettier now that 4nm 40XX series cards are available. Hell, an Intel A770 with 16gb of VRAM is still cheaper than adding 16gb of RAM to a Mac Mini. It's good stuff, but pitched a bit hard with all the marketing. From where I'm standing, it looks like Apple is playing catch-up with their GPUs and acceleration tech. tentatively removes hat of jaded realism

Nevermark 4y ago

Given that the RAM is directly accessible by CPU, GPU and DPU/Neural cores, it really is premium RAM. Apple's visionary hardware team has finally caught up with Apple's visionary high RAM prices! :)

DennisAleynikov 4y ago
Also um... Idk if anyone cares but apple didn't do anything to make unified memory. All arm chips do this lmao Apple just has the best known arm chips with the highest mobile performance (yes faster server arm chips exist too)
- fooker 4y ago
  It’s amazing how Apple “doesn’t do” anything, but manages to define industry trends over and over again!
  - 7to2 4y ago
    That's what brand recognition and brand loyalty (and the money they generate) allows you to do. This is not necessarily a bad thing since it forces other competitors - who like to artificially limit their products - to actually get of their greedy asses and compete.
    - DennisAleynikov 4y ago
      Yeah which is why... Unified memory on Chromebooks and android devices as well as surface pro serve the same purpose. It's marketed as an upsell by apple because it's an advantage of ARM but many people will never care to learn what that means

deleted 4y ago

[deleted]