16 ms·

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread t

Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...

ok123456 4y ago
You can mmap from python.
- westurner 4y ago
  The CPython mmap module docs: https://docs.python.org/3/library/mmap.html https://docs.python.org/3/library/mmap.html zero_buffer (CFFI, 2013) https://github.com/alex/zero_buffer/blob/master/zero_buffer.py https://github.com/alex/zero_buffer/blob/master/zero_buffer.... "Buffers on the edge: Python and Rust" (2022) https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ : > If you have a Python object and want to obtain its buffer, you can do so with memoryview in Python or PyObject_GetBuffer in C. If you’re defining a class and want to expose a buffer, you can do so in Python by… actually you can’t, only classes implemented in C can implement the buffer protocol. To implement the buffer protocol in C, you provide the bf_getbuffer and bf_releasebuffer functions which are called to obtain a buffer from an object and when that buffer is being released, respectively. iocursor (CPython C API, ~Rust std::io::Cursor) https://github.com/althonos/iocursor https://github.com/althonos/iocursor Arrow Python (C++) > On disk and MemoryMappedFile s: https://arrow.apache.org/docs/python/memory.html#on-disk-and-memory-mapped-files https://arrow.apache.org/docs/python/memory.html#on-disk-and... "Apache Arrow: Read DataFrame With Zero Memory" (2020) https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a https://towardsdatascience.com/apache-arrow-read-dataframe-w... pyarrow.Tensor: https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor https://arrow.apache.org/docs/python/generated/pyarrow.Tenso... ONNX is built on protocolbuffers/protobufs (google/protobufs), while Arrow is built on google/flatbuffers. FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers : > It supports “zero-copy” deserialization, so that accessing the serialized data does not require first copying it into a separate part of memory. This makes accessing data in these formats much faster than data in formats requiring more extensive processing, such as JSON, CSV, and in many cases Protocol Buffers. Compared to other serialization formats however, the handling of FlatBuffers requires usually more code, and some operations are not possible (like some mutation operations).
- lostmsu 4y ago
  In fact, you can mmap from PyTorch directly.
  - ok123456 4y ago
    and numpy
MontyCarloHall 4y ago
>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that much. All of the Python numeric computing frameworks (Numpy, PyTorch, TensorFlow, etc.) are basically just wrappers for lower level C++/C/Fortran code. Unless you’re doing something boneheaded and converting framework-native tensors to Python objects, passing tensors around within a framework essentially just passes a pointer around, which has marginal overhead even when encapsulated in a bloated Python object. Indeed, a huge number of PyTorch operations are explicitly zero copy: https://pytorch.org/docs/stable/tensor_view.html https://pytorch.org/docs/stable/tensor_view.html
- oceanplexian 4y ago
  It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.
  - eastWestMath 4y ago
    I was in a PLT group in grad school going into robotics. I could spend all day ranting about how Python is just completely unsuitable for professional software development. Even something like F# would be an enormous improvement.
  - rfoo 4y ago
    The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flexibility and iteration speed naturally, just like why people use Rails. In fact I'd love to see that Transformer really dominates. We can then start to converge on software. And compute-wise transformers are really simple, too!
catchnear4321 4y ago
Money did. Why waste developer hours (meaning effort) if you can just scale the infra for a little cash? Do it in small enough increments and the increases only outweigh FTEs if you consider all scaling events and look at a long enough time scale. Suddenly it takes way too much for way too little, but it cost half as many overpaid developers who can’t be arsed to performance. Edit: in case that sounds like the opposite of intended, ggerganov and jart are the outliers, the exception.
shakow 4y ago
> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.
- xmprt 4y ago
  This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.
  - jart 4y ago
    Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who are supposed to be solving them would never even notice; in fact, they'd probably think you're trolling them if you ever brought it up.
    - axlee 4y ago
      Tragedy of the commons only work for things you don't directly pay for.
      - nablags 4y ago
        well in a way - open source software something that you don’t directly pay for
        sli 4y ago
        More saliently, the overwhelming majority of the Linux kernel's direct and extended userbase has contributed nothing at all directly to the Linux kernel, as just one example.
gct 4y ago
This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.
- ChatPGT 4y ago
  >> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
  - sn_master 4y ago
    I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
    - Swizec 4y ago
      Python has the ecosystem. That’s it. The lingua franca of data science. At this point it doesn’t even matter anymore why. Just like you’re not gonna usurp JavaScript on the web.
    - 6510 4y ago
      Before an NDA send him to Rura Penthe I use to have an internet friend pedantic about seemingly useless compilers and interpreters. Quests like: use obscure language A to translate obscure language B to obscure language C. Then use B compiled to C to interpret D. A long story short, in the future the AI can just convert all our code to FORTH or HolyC or some "creative" combination of languages chosen by prophecy (read: hallucination) perhaps even Python — as a show of strength.
    - codexon 4y ago
      There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.
rfoo 4y ago
Sigh. It's not like the zero copy buzzword is going to help you during training, all your weights have to stay on GPU, you are going to sample your training data randomly and your data is on a networked storage anyway, so mmap HURTS. You'd better just O_DIRECT. Similarly, as long as you run your inference on GPU it's not like you can mmap... And I have indeed worked on inference runtimes for mobile devices and on the rare cases we need to use CPU only (hey, your phone also have GPUs since forever) at $PREVIOUS_JOB we did have a mmap-able model format, it also helps in TEE/SGX/whatever enclave tech. Oh, and there are no Python at all. The recent development of ggml is interesting as it catches a moment that "big ML shop infra" guys don't care: running models on Apple Silicon. M1/M2s are expensive enough that we don't consider deploying them instead of those 1000000000 bizarre accelerators in production, yet everyone on HN seems to have one and hey it's fast enough for LMs. They are rather unique as they are CPU+high bandwidth RAM+accelerators with totally shared RAM with CPU, instead of some GPU shit. tldr it's not like "big ML shop infra" guys are stupid and leaves performance on table. They just don't run their production workload on MacBooks. That's where the community shine right?
- sroussey 4y ago
  On a Mac, mmap definitely works for the GPU since it’s all the same unified memory.
- miraculixx 4y ago
  This! Thank you.
- danbst 4y ago
  in llama.cpp inference runs on CPU, using AVX-2 optimizations. You don't need GPU at all It runs on my 2015 ThinkPad!
d0mine 4y ago
It is easy to use mmap from Python. You can do zero copy too e.g., https://stackoverflow.com/questions/17244488/reading-struct-in-python-from-created-struct-in-c https://stackoverflow.com/questions/17244488/reading-struct-... (see mmap+frombuffer example) Though in practice, in many cases, mmap won't be faster, it can be even slower than open+read.
somesortofsystm 4y ago
> such clever use of mmap Just wanna say, that this use of mmap() is cleverly used in this context, but should be acknowledged as a widely accepted industry standard practice for getting higher performance, particularly in embedded applications but also in performance-oriented apps such as digital audio workstations, video editing systems, and so on.
- jart 4y ago
  Just because mmap() is commonly used doesn't mean it's commonly understood. Yes, it powers just about everything important in terms of the skeletons of our local systems. So why has the thought of using it occurred to so few people until now? Almost a whole generation has passed since things like mmap() were relegated to "the work's been done!" category of computing. People moved on to caring about things like My Browser and The Cloud where mmap() doesn't exist. Most people don't know about it. The ones who do, are reluctant to use it. Scientific computing projects are totally devoted to supporting MSVC (since you just know data scientists are secretly using those GPUs for gaming) so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32 before any chance to fully consider the true depth of its value would kick in. Plus data migrations are very difficult to pull off. It worked here due to the outpouring of community support, since people were blocked on this. But for a corporation with tons of cash to burn, it's a harder sell.
  - somesortofsystm 4y ago
    The Cloud has been with us since the birth of computing. What is happening is, the computing industry goes through waves of attrition, whereby the schools push everyone up the Brand New Stack, while industry, frustrated with generations of programmers who can't program, just Builds Another Stack. Repeat, ad infinitum. In the cracks you'll find people re-learning things they should've known, if only they weren't slagging off the grey beards .. or, even worse .. as grey beards not paying attention to the discoveries of youth. >Most people don't know about it. The ones who do, are reluctant to use it. Not so sure about this. The reluctance is emotional, its not technical. Nobody is killing POSIX under all of this - it is deployed. Therefore, learn it. >so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32 Does not compute. Own up, you're an AI.
a-dub 4y ago
you'd be surprised how many professional programmers these days work exclusively in high level languages and know nothing about using operating system features to their fullest. but to your point, until technology itself actually replaces us, deeply skilled computer people are always going to be able to squeeze more performance out of software implemented in high level languages by those who have not studied computers extensively.
deadly_syn 4y ago
Thank you for saying it out loud, I thought I was going crazy!
amrb 4y ago
For the life of me I could never fix torch.load, they'll say just quantization (convert) a model to 4/8bit to make it smaller but you'll get crashes when out of system memory plus no docs.. then you admit defeat by using more swapfile :S

ok123456 4y ago

You can mmap from python.

westurner 4y ago
The CPython mmap module docs: https://docs.python.org/3/library/mmap.html https://docs.python.org/3/library/mmap.html zero_buffer (CFFI, 2013) https://github.com/alex/zero_buffer/blob/master/zero_buffer.py https://github.com/alex/zero_buffer/blob/master/zero_buffer.... "Buffers on the edge: Python and Rust" (2022) https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ : > If you have a Python object and want to obtain its buffer, you can do so with memoryview in Python or PyObject_GetBuffer in C. If you’re defining a class and want to expose a buffer, you can do so in Python by… actually you can’t, only classes implemented in C can implement the buffer protocol. To implement the buffer protocol in C, you provide the bf_getbuffer and bf_releasebuffer functions which are called to obtain a buffer from an object and when that buffer is being released, respectively. iocursor (CPython C API, ~Rust std::io::Cursor) https://github.com/althonos/iocursor https://github.com/althonos/iocursor Arrow Python (C++) > On disk and MemoryMappedFile s: https://arrow.apache.org/docs/python/memory.html#on-disk-and-memory-mapped-files https://arrow.apache.org/docs/python/memory.html#on-disk-and... "Apache Arrow: Read DataFrame With Zero Memory" (2020) https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a https://towardsdatascience.com/apache-arrow-read-dataframe-w... pyarrow.Tensor: https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor https://arrow.apache.org/docs/python/generated/pyarrow.Tenso... ONNX is built on protocolbuffers/protobufs (google/protobufs), while Arrow is built on google/flatbuffers. FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers : > It supports “zero-copy” deserialization, so that accessing the serialized data does not require first copying it into a separate part of memory. This makes accessing data in these formats much faster than data in formats requiring more extensive processing, such as JSON, CSV, and in many cases Protocol Buffers. Compared to other serialization formats however, the handling of FlatBuffers requires usually more code, and some operations are not possible (like some mutation operations).
lostmsu 4y ago
In fact, you can mmap from PyTorch directly.
- ok123456 4y ago
  and numpy

MontyCarloHall 4y ago

>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that much. All of the Python numeric computing frameworks (Numpy, PyTorch, TensorFlow, etc.) are basically just wrappers for lower level C++/C/Fortran code. Unless you’re doing something boneheaded and converting framework-native tensors to Python objects, passing tensors around within a framework essentially just passes a pointer around, which has marginal overhead even when encapsulated in a bloated Python object. Indeed, a huge number of PyTorch operations are explicitly zero copy: https://pytorch.org/docs/stable/tensor_view.html https://pytorch.org/docs/stable/tensor_view.html

oceanplexian 4y ago
It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.
- eastWestMath 4y ago
  I was in a PLT group in grad school going into robotics. I could spend all day ranting about how Python is just completely unsuitable for professional software development. Even something like F# would be an enormous improvement.
- rfoo 4y ago
  The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flexibility and iteration speed naturally, just like why people use Rails. In fact I'd love to see that Transformer really dominates. We can then start to converge on software. And compute-wise transformers are really simple, too!

catchnear4321 4y ago

Money did. Why waste developer hours (meaning effort) if you can just scale the infra for a little cash? Do it in small enough increments and the increases only outweigh FTEs if you consider all scaling events and look at a long enough time scale. Suddenly it takes way too much for way too little, but it cost half as many overpaid developers who can’t be arsed to performance. Edit: in case that sounds like the opposite of intended, ggerganov and jart are the outliers, the exception.

shakow 4y ago

> Perhaps SWE is dead after all, but LLMs didn't kill it... Cheap electronics did. 32GB of RAM is maybe $150, a developer converting & maintaining your system to use mmap is $150k/year.

xmprt 4y ago
This still doesn't make sense. It doesn't take a full year to do optimizations like this. Maybe a month at most if you include the investigation time. And the memory usage is $150 times the number of users which is in the thousands at least.
- jart 4y ago
  Tragedy of the commons. If you want to do something that benefits everyone a little bit, and you can't productize it like OpenAI's $20/month subscription, then there's no rational economic reason to do it, and you have to wait for someone like me who has an irrational love of coding. It's not a lifestyle that makes you rich, but it does help you see the opportunities to fix problems that the well-resourced folks who are supposed to be solving them would never even notice; in fact, they'd probably think you're trolling them if you ever brought it up.
  - axlee 4y ago
    Tragedy of the commons only work for things you don't directly pay for.
    - nablags 4y ago
      well in a way - open source software something that you don’t directly pay for
      - sli 4y ago
        More saliently, the overwhelming majority of the Linux kernel's direct and extended userbase has contributed nothing at all directly to the Linux kernel, as just one example.

gct 4y ago

This doesn't even seem that clever, just regular ol' use of mmap where there was none before. Wonder what other performance is being left on the floor. I'm convinced entire power plants could be retired if the world stopped using python unfortunately.

ChatPGT 4y ago
>> I'm convinced entire power plants could be retired if the world stopped using python unfortunately. On the other hand, many business and professionals wouldn't exist :)
- sn_master 4y ago
  I can't find a single good argument for Python based on merit that's not at least 15+ years dated and stems from "But Google is using it". It's not the easiest syntax, not the best compiler support, performance and threading is a joke. The entire language is based on hype back from the time when the only two mainstream languages were C++ and Java.
  - Swizec 4y ago
    Python has the ecosystem. That’s it. The lingua franca of data science. At this point it doesn’t even matter anymore why. Just like you’re not gonna usurp JavaScript on the web.
  - 6510 4y ago
    Before an NDA send him to Rura Penthe I use to have an internet friend pedantic about seemingly useless compilers and interpreters. Quests like: use obscure language A to translate obscure language B to obscure language C. Then use B compiled to C to interpret D. A long story short, in the future the AI can just convert all our code to FORTH or HolyC or some "creative" combination of languages chosen by prophecy (read: hallucination) perhaps even Python — as a show of strength.
  - codexon 4y ago
    There were plenty of other languages competing with python for the same niche such as perl, ruby, js, php etc... Python is superior to all of those just for syntax alone, it is easier and cleaner to both read and write.

rfoo 4y ago

Sigh. It's not like the zero copy buzzword is going to help you during training, all your weights have to stay on GPU, you are going to sample your training data randomly and your data is on a networked storage anyway, so mmap HURTS. You'd better just O_DIRECT. Similarly, as long as you run your inference on GPU it's not like you can mmap... And I have indeed worked on inference runtimes for mobile devices and on the rare cases we need to use CPU only (hey, your phone also have GPUs since forever) at $PREVIOUS_JOB we did have a mmap-able model format, it also helps in TEE/SGX/whatever enclave tech. Oh, and there are no Python at all. The recent development of ggml is interesting as it catches a moment that "big ML shop infra" guys don't care: running models on Apple Silicon. M1/M2s are expensive enough that we don't consider deploying them instead of those 1000000000 bizarre accelerators in production, yet everyone on HN seems to have one and hey it's fast enough for LMs. They are rather unique as they are CPU+high bandwidth RAM+accelerators with totally shared RAM with CPU, instead of some GPU shit. tldr it's not like "big ML shop infra" guys are stupid and leaves performance on table. They just don't run their production workload on MacBooks. That's where the community shine right?

sroussey 4y ago
On a Mac, mmap definitely works for the GPU since it’s all the same unified memory.
miraculixx 4y ago
This! Thank you.
danbst 4y ago
in llama.cpp inference runs on CPU, using AVX-2 optimizations. You don't need GPU at all It runs on my 2015 ThinkPad!

d0mine 4y ago

It is easy to use mmap from Python. You can do zero copy too e.g., https://stackoverflow.com/questions/17244488/reading-struct-in-python-from-created-struct-in-c https://stackoverflow.com/questions/17244488/reading-struct-... (see mmap+frombuffer example) Though in practice, in many cases, mmap won't be faster, it can be even slower than open+read.

somesortofsystm 4y ago

> such clever use of mmap Just wanna say, that this use of mmap() is cleverly used in this context, but should be acknowledged as a widely accepted industry standard practice for getting higher performance, particularly in embedded applications but also in performance-oriented apps such as digital audio workstations, video editing systems, and so on.

jart 4y ago
Just because mmap() is commonly used doesn't mean it's commonly understood. Yes, it powers just about everything important in terms of the skeletons of our local systems. So why has the thought of using it occurred to so few people until now? Almost a whole generation has passed since things like mmap() were relegated to "the work's been done!" category of computing. People moved on to caring about things like My Browser and The Cloud where mmap() doesn't exist. Most people don't know about it. The ones who do, are reluctant to use it. Scientific computing projects are totally devoted to supporting MSVC (since you just know data scientists are secretly using those GPUs for gaming) so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32 before any chance to fully consider the true depth of its value would kick in. Plus data migrations are very difficult to pull off. It worked here due to the outpouring of community support, since people were blocked on this. But for a corporation with tons of cash to burn, it's a harder sell.
- somesortofsystm 4y ago
  The Cloud has been with us since the birth of computing. What is happening is, the computing industry goes through waves of attrition, whereby the schools push everyone up the Brand New Stack, while industry, frustrated with generations of programmers who can't program, just Builds Another Stack. Repeat, ad infinitum. In the cracks you'll find people re-learning things they should've known, if only they weren't slagging off the grey beards .. or, even worse .. as grey beards not paying attention to the discoveries of youth. >Most people don't know about it. The ones who do, are reluctant to use it. Not so sure about this. The reluctance is emotional, its not technical. Nobody is killing POSIX under all of this - it is deployed. Therefore, learn it. >so any thought devs may have had previously about using mmap() would have certainly triggered fears w.r.t. WIN32 Does not compute. Own up, you're an AI.

a-dub 4y ago

you'd be surprised how many professional programmers these days work exclusively in high level languages and know nothing about using operating system features to their fullest. but to your point, until technology itself actually replaces us, deeply skilled computer people are always going to be able to squeeze more performance out of software implemented in high level languages by those who have not studied computers extensively.

deadly_syn 4y ago

Thank you for saying it out loud, I thought I was going crazy!

amrb 4y ago

For the life of me I could never fix torch.load, they'll say just quantization (convert) a model to 4/8bit to make it smaller but you'll get crashes when out of system memory plus no docs.. then you admit defeat by using more swapfile :S