3 ms·

Could in-memory compression used to bring down the RAM requirements? There are some performance compressors like Blosch tuned for this: https://www.blosc.org/

Could in-memory compression used to bring down the RAM requirements? There are some performance compressors like Blosch tuned for this: https://www.blosc.org/pages/blosc-in-depth/ https://www.blosc.org/pages/blosc-in-depth/ “Faster than memcpy” is the slogan.

addaon 4y ago
MacOS has transparent memory compression. Unclear to me if that's made its way to iPhone, but if it hasn't yet it will sooner or later.
- comboy 4y ago
  Memory compression? I can't find any good resources to read about it, any hints? I'm having trouble imaging how could it possibly work without totally destroying performance.
  - miohtama 4y ago
    On Mac, you can find Compressed memory in Activity monitor. It’s something similar to swap - apps do not need to have built in support for it.
    - flatiron 4y ago
      It segments a certain amount of ram to “swap” to which means compress and store. Normal blue sky ram operations are not compressed on macOS
    - comboy 4y ago
      Oh, yes compressed swap makes much more sense, thanks.
      - kergonath 4y ago
        It is not compressed swap, the compressed data is still in RAM. The OS just compresses inactive memory, with a couple of criteria to define “inactive”.
    - smcleod 4y ago
      Many operations are actually a lot faster with compressed memory than without. It's all about where the bottleneck is.
  - kccqzy 4y ago
    It doesn't destroy performance for the simple reason that nowadays memory access is slower than pure compute. If you need to use compute to produce some data to be stored in memory, your overall throughput could very well be faster than without compression. There have been a large amount of innovation on fast compression and fast decompression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms: * lz4: https://lz4.github.io/lz4/ https://lz4.github.io/lz4/ * Google's snappy: https://github.com/google/snappy https://github.com/google/snappy * Facebook's zstd in fast mode: http://facebook.github.io/zstd/#benchmarks http://facebook.github.io/zstd/#benchmarks
- miohtama 4y ago
  My guess is that iPhone is purely “kill app” instead of “compress memory / swap” OOM model. This makes more sense for mobile.
  - musicale 4y ago
    IIRC from WWDC they said that inactive/suspended apps get their memory compressed to free up memory for the current active/foreground app. Seems to mesh well with the iOS idea of using a single app at a time and minimizing background processing in apps that you aren't actively using. In an out of memory situation I think apps just get killed as you suggest.
  - Sirened 4y ago
    iOS uses memory compression but not swap. iOS devices actually have special CPU instructions to speed up compression of up to page size increments specifically to aid in this model [1] [1] https://github.com/apple-oss-distributions/xnu/blob/bb611c8fecc755a0d8e56e2fa51513527c5b7a0e/osfmk/arm64/WKdmCompress_16k.s https://github.com/apple-oss-distributions/xnu/blob/bb611c8f...
- astrange 4y ago
  Memory compression is a generalization of swap, which is only for dynamic memory; files on disk don't need it because you can just read them off the disk. The problem is that GPUs don't support virtual memory paging, so they can't read files nor decompress nor swap anything unless you write it yourself, which is a lot slower. Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing!
  - earthscienceman 4y ago
    Wait. This comment just blew my mind. Does that imply that you might be able to measure the efficiency of a model by it's compressibility? Note, I'm trying to recognize efficient and accurate are not the same. One could imagine evaluating a model on a 2d performance and compression map somehow.
  - ColonelPhantom 4y ago
    > Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing! I feel like they're kind of two sides of the same coin: learning is about putting more information in the same data, while compression is about putting the same information in less data. I'm wondering if some lossy floating-point compressor (such as zfp) would work.
    - astrange 4y ago
      > I'm wondering if some lossy floating-point compressor (such as zfp) would work. Well apparently this can work; StableDiffusion comes as 32-bit and 16-bit float versions. I'm kind of surprised they both work, but that's lossy compression.
      - ColonelPhantom 4y ago
        Sure, but 16-bit float is pretty primitive compression, as it does not exploit any redundancy in the input. zfp groups numbers together in chunks, which means that correlated numbers can be represented more precisely. Its algorithm is described here: https://zfp.readthedocs.io/en/release1.0.0/algorithm.html#lossy-compression https://zfp.readthedocs.io/en/release1.0.0/algorithm.html#lo... I would like to see if the zfp can be applied to something like Stable Diffusion (or other ML models) and give better results than regular floats at the same size.
conradev 4y ago
In-memory compression means the memory is inherently dirty memory On Apple platforms if you mmap a read-only file into the process address space, then it is "clean" memory. It is clean because the kernel can drop it at any time because it already exists on disk. You essentially can offload the memory management to the kernel page cache. The downside is that if you run up to the limit and the "working set" can't fit entirely in memory, then you run into page faults which incur an I/O cost. The advantage is that the kernel will drop the page cache before it considers killing your process to reclaim memory. That said, I don't know the typical access patterns for neural network inference, so I don't know how the page faults would effect performance
Scaevolus 4y ago
No. The memory usage is due to huge series of floating point numbers without much redundancy that you could squeeze out with a compressor.
tehjoker 4y ago
zfp compression might be an interesting thing to try. In fixed rate mode, it supports random access too.
liuliu 4y ago
Should be doable for parameters but at that point, you don't need compression rather just LLM.Int8 tricks would be sufficient. For activations, I wrote about it a while back: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-deep-neural-network-training-over-mixed-precision-with-tensor-compression/ https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de... It is not as useful for this case (inference) because the activations holds long (UNet holds downsampling passes' activations and use that for upsampling) is not that much of a memory (in the range of a few megabytes). If it is for training, it is probably more useful.

addaon 4y ago

MacOS has transparent memory compression. Unclear to me if that's made its way to iPhone, but if it hasn't yet it will sooner or later.

comboy 4y ago
Memory compression? I can't find any good resources to read about it, any hints? I'm having trouble imaging how could it possibly work without totally destroying performance.
- miohtama 4y ago
  On Mac, you can find Compressed memory in Activity monitor. It’s something similar to swap - apps do not need to have built in support for it.
  - flatiron 4y ago
    It segments a certain amount of ram to “swap” to which means compress and store. Normal blue sky ram operations are not compressed on macOS
  - comboy 4y ago
    Oh, yes compressed swap makes much more sense, thanks.
    - kergonath 4y ago
      It is not compressed swap, the compressed data is still in RAM. The OS just compresses inactive memory, with a couple of criteria to define “inactive”.
  - smcleod 4y ago
    Many operations are actually a lot faster with compressed memory than without. It's all about where the bottleneck is.
- kccqzy 4y ago
  It doesn't destroy performance for the simple reason that nowadays memory access is slower than pure compute. If you need to use compute to produce some data to be stored in memory, your overall throughput could very well be faster than without compression. There have been a large amount of innovation on fast compression and fast decompression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms: * lz4: https://lz4.github.io/lz4/ https://lz4.github.io/lz4/ * Google's snappy: https://github.com/google/snappy https://github.com/google/snappy * Facebook's zstd in fast mode: http://facebook.github.io/zstd/#benchmarks http://facebook.github.io/zstd/#benchmarks
miohtama 4y ago
My guess is that iPhone is purely “kill app” instead of “compress memory / swap” OOM model. This makes more sense for mobile.
- musicale 4y ago
  IIRC from WWDC they said that inactive/suspended apps get their memory compressed to free up memory for the current active/foreground app. Seems to mesh well with the iOS idea of using a single app at a time and minimizing background processing in apps that you aren't actively using. In an out of memory situation I think apps just get killed as you suggest.
- Sirened 4y ago
  iOS uses memory compression but not swap. iOS devices actually have special CPU instructions to speed up compression of up to page size increments specifically to aid in this model [1] [1] https://github.com/apple-oss-distributions/xnu/blob/bb611c8fecc755a0d8e56e2fa51513527c5b7a0e/osfmk/arm64/WKdmCompress_16k.s https://github.com/apple-oss-distributions/xnu/blob/bb611c8f...
astrange 4y ago
Memory compression is a generalization of swap, which is only for dynamic memory; files on disk don't need it because you can just read them off the disk. The problem is that GPUs don't support virtual memory paging, so they can't read files nor decompress nor swap anything unless you write it yourself, which is a lot slower. Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing!
- earthscienceman 4y ago
  Wait. This comment just blew my mind. Does that imply that you might be able to measure the efficiency of a model by it's compressibility? Note, I'm trying to recognize efficient and accurate are not the same. One could imagine evaluating a model on a 2d performance and compression map somehow.
- ColonelPhantom 4y ago
  > Also, ML models (probably) can't be compressed because they already are compressed; learning and compression are the same thing! I feel like they're kind of two sides of the same coin: learning is about putting more information in the same data, while compression is about putting the same information in less data. I'm wondering if some lossy floating-point compressor (such as zfp) would work.
  - astrange 4y ago
    > I'm wondering if some lossy floating-point compressor (such as zfp) would work. Well apparently this can work; StableDiffusion comes as 32-bit and 16-bit float versions. I'm kind of surprised they both work, but that's lossy compression.
    - ColonelPhantom 4y ago
      Sure, but 16-bit float is pretty primitive compression, as it does not exploit any redundancy in the input. zfp groups numbers together in chunks, which means that correlated numbers can be represented more precisely. Its algorithm is described here: https://zfp.readthedocs.io/en/release1.0.0/algorithm.html#lossy-compression https://zfp.readthedocs.io/en/release1.0.0/algorithm.html#lo... I would like to see if the zfp can be applied to something like Stable Diffusion (or other ML models) and give better results than regular floats at the same size.

conradev 4y ago

In-memory compression means the memory is inherently dirty memory On Apple platforms if you mmap a read-only file into the process address space, then it is "clean" memory. It is clean because the kernel can drop it at any time because it already exists on disk. You essentially can offload the memory management to the kernel page cache. The downside is that if you run up to the limit and the "working set" can't fit entirely in memory, then you run into page faults which incur an I/O cost. The advantage is that the kernel will drop the page cache before it considers killing your process to reclaim memory. That said, I don't know the typical access patterns for neural network inference, so I don't know how the page faults would effect performance

Scaevolus 4y ago

No. The memory usage is due to huge series of floating point numbers without much redundancy that you could squeeze out with a compressor.

tehjoker 4y ago

zfp compression might be an interesting thing to try. In fixed rate mode, it supports random access too.

liuliu 4y ago

Should be doable for parameters but at that point, you don't need compression rather just LLM.Int8 tricks would be sufficient. For activations, I wrote about it a while back: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-deep-neural-network-training-over-mixed-precision-with-tensor-compression/ https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de... It is not as useful for this case (inference) because the activations holds long (UNet holds downsampling passes' activations and use that for upsampling) is not that much of a memory (in the range of a few megabytes). If it is for training, it is probably more useful.