5 ms·

It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-source-sol

It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-source-solution-for-cloning-chatgpt-with-a-complete-rlhf-pipeline-5edf08fb538b https://medium.com/@yangyou_berkeley/colossalchat-an-open-so... Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...

SebJansen 4y ago
does the 1gb limit extend to wasm?
- jesse__ 4y ago
  WASM is specified to have 32-bit pointers, which is 4GB. AFAIK browser implementations respect that (when I did some nominal testing a couple years ago)
jesse__ 4y ago
I thought the memory limit (in V8 at least) was 2GB due to the GC not wanting to pass 64 bit pointers around, and using the high bit of a 32-bit offset for .. something I now forget ..? Do you have a source showing a JS runtime with a 1GB limit?
- jesse__ 4y ago
  UPDATE: After a nominal amount of googling around it appears valid sizes have increased on 64-bit systems to a maximum of 8GB, and stayed at 2GB on 32-bit systems, for FF at least. I guess it's probably 'implementation defined' https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer#resizing_arraybuffers https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
lmeyerov 4y ago
Interesting, since I looked last year, Chrome has started raising the caps internally on buffer allocation to potentially 16GB: https://chromium.googlesource.com/chromium/src/+/2bf3e35d7a4cf9286092bd2f7e95daf332d1cea2 https://chromium.googlesource.com/chromium/src/+/2bf3e35d7a4... Last time I tried on a few engines, it was just 1-2GB for typed arrays, which are essentially the backing structure for this kind of work. Be interesting to try again.. For our product, we actually want to dump 10GB+ on to the WebGL side, which may or may not get mirrored on the CPU side. Not sure if additional limits there on the software side. And after that, consumer devices often have another 10GB+ CPU RAM free, which we'd also like to use for our more limited non-GPU stuff :)

SebJansen 4y ago

does the 1gb limit extend to wasm?

jesse__ 4y ago
WASM is specified to have 32-bit pointers, which is 4GB. AFAIK browser implementations respect that (when I did some nominal testing a couple years ago)

jesse__ 4y ago

I thought the memory limit (in V8 at least) was 2GB due to the GC not wanting to pass 64 bit pointers around, and using the high bit of a 32-bit offset for .. something I now forget ..? Do you have a source showing a JS runtime with a 1GB limit?

jesse__ 4y ago
UPDATE: After a nominal amount of googling around it appears valid sizes have increased on 64-bit systems to a maximum of 8GB, and stayed at 2GB on 32-bit systems, for FF at least. I guess it's probably 'implementation defined' https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer#resizing_arraybuffers https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

lmeyerov 4y ago

Interesting, since I looked last year, Chrome has started raising the caps internally on buffer allocation to potentially 16GB: https://chromium.googlesource.com/chromium/src/+/2bf3e35d7a4cf9286092bd2f7e95daf332d1cea2 https://chromium.googlesource.com/chromium/src/+/2bf3e35d7a4... Last time I tried on a few engines, it was just 1-2GB for typed arrays, which are essentially the backing structure for this kind of work. Be interesting to try again.. For our product, we actually want to dump 10GB+ on to the WebGL side, which may or may not get mirrored on the CPU side. Not sure if additional limits there on the software side. And after that, consumer devices often have another 10GB+ CPU RAM free, which we'd also like to use for our more limited non-GPU stuff :)