4 ms·

I'd agree if Apple were in the business of making datacentre infrastructure. Nobody is running these large scale models on their personal devices. Sure, some

by TOMDM 4y ago

I'd agree if Apple were in the business of making datacentre infrastructure. Nobody is running these large scale models on their personal devices. Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.

earthboundkid 4y ago
Someone has to pay for that compute power. There are returns to scale and savings from sharing a machine when it would be idling, but those aren’t a factor of 10x. The AI will get in your pocket eventually and they’ll be ready to charge a premium for it.
somenameforme 4y ago
GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems. It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). [1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B [2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577
- TOMDM 4y ago
  Apples hardware, and 4 RTX 3090's being able to run a GPT scale model quantized to 4 byte ints are worlds apart. Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant. I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
  - earthboundkid 4y ago
    Isn’t the research now on using float8? Anyway, it will take time to move battleship, but they’re well aligned and motivated. Their UI team has gone to hell in recent years, but hardware has been strong with nothing to waste the power on. Now there is something to use it for.
    - somenameforme 4y ago
      Are we not putting the cart before the horse? Where's the widespread usage for this that people would actually want? I can think of quite a lot of uses, but oddly enough most are quite negative and not widespread in any case. Media companies can use it to fire employees and increase profit margins. Hollywood can use it to rapidly churn out even more overtly derivative and generic movies. Students can use it to cheat on papers. Governments, and marketing agencies alike, can use it for endless contextually sensitive propaganda on social media. Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic. I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.
jsemrau 4y ago
Not saying anything about training those models, but I can run the weights of Stable Diffusion without larger problems on a vintage RTX 1080Ti.

by TOMDM 4y ago

earthboundkid 4y ago

Someone has to pay for that compute power. There are returns to scale and savings from sharing a machine when it would be idling, but those aren’t a factor of 10x. The AI will get in your pocket eventually and they’ll be ready to charge a premium for it.

somenameforme 4y ago

GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems. It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). [1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B [2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577

TOMDM 4y ago
Apples hardware, and 4 RTX 3090's being able to run a GPT scale model quantized to 4 byte ints are worlds apart. Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant. I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
- earthboundkid 4y ago
  Isn’t the research now on using float8? Anyway, it will take time to move battleship, but they’re well aligned and motivated. Their UI team has gone to hell in recent years, but hardware has been strong with nothing to waste the power on. Now there is something to use it for.
  - somenameforme 4y ago
    Are we not putting the cart before the horse? Where's the widespread usage for this that people would actually want? I can think of quite a lot of uses, but oddly enough most are quite negative and not widespread in any case. Media companies can use it to fire employees and increase profit margins. Hollywood can use it to rapidly churn out even more overtly derivative and generic movies. Students can use it to cheat on papers. Governments, and marketing agencies alike, can use it for endless contextually sensitive propaganda on social media. Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic. I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.

jsemrau 4y ago

Not saying anything about training those models, but I can run the weights of Stable Diffusion without larger problems on a vintage RTX 1080Ti.