5 ms·

Memory usage is an interesting problem, because the failure mode is so much more painful (crashes and complete lock-ups) than high CPU usage. And even if you r

Memory usage is an interesting problem, because the failure mode is so much more painful (crashes and complete lock-ups) than high CPU usage. And even if you return memory to the OS, that doesn't actually solve too-high memory usage. Some things you can do: The article mentions `__slots__` for reducing object memory use, and other approaches include just having fewer objects: for example, a dict of lists uses far less memory than a list of dicts with repeating fields. And you can also in many cases switch to a dataframe with Pandas, saving even more memory (https://pythonspeed.com/articles/python-object-memory/ https://pythonspeed.com/articles/python-object-memory/ covers all of those). For numeric data, a NumPy array gets rid of the per-integer overhead for Python, so a Python list of numbers use way more memory than an equivalent NumPy array (https://pythonspeed.com/articles/python-integers-memory/ https://pythonspeed.com/articles/python-integers-memory/).

ublaze 6y ago
I've also seen some tricks like stripping docstrings from the release since those strings (hopefully) aren't used in production.
- itamarst 6y ago
  I can't imagine that has any significant impact on memory usage, at least for normal multi-GB-RAM servers: the docstrings will only be loaded once.
- tomwojcik 6y ago
  That's nothing. Armin Ronacher was complaining somewhere about the performance hit from static typing in python and i never though about that before. It'd be best if all of that was removed in prod build along with unused imports, kind of what TypeScript does.
  - ublaze 6y ago
    imports can't be safely automatically stripped in Python (or even 100% safely re-ordered) since they could have side effects. We run a linter that confirms with the user before stripping them, which works out well in practice.
    - aitchnyu 6y ago
      Its apparently true for RPython too, where side effects which cause "methods and other class attributes do not change after startup".
  - aaronbwebber 6y ago
    like PEP484 type annotations having a runtime performance hit? I would be very interested in seeing a link to this if you can dig it up.
  - orf 6y ago
    There used to be a performance hit because the annotation objects where all instantiated and attached to the function objects. This is no longer the case - they are only instantiated when inspected.
user5994461 6y ago
Another thing you can do is to deduplicate strings in memory. got 17% memory reduction for loading json objects with some repeated values. https://thehftguy.com/2020/09/25/how-to-deduplicate-string-objects-in-memory-in-python/ https://thehftguy.com/2020/09/25/how-to-deduplicate-string-o...
joshlk 6y ago
DataClassFrames also takes the approach of storing lists of dataclasses as a “dataclass of arrays” - know as data orientated design: https://gitHub.com/joshlk/dataclassframe https://gitHub.com/joshlk/dataclassframe (Full disclosure I’m the author of the project)
- chubot 6y ago
  FWIW your comment caught my interest as a longstanding user of data frames and Python [1]. But I found the README quite confusing. I guess it's a new project so that's understandable. Why use DataClassFrames and not pandas? Because it's statically typed? The comparison table puts pandas and dataclassframes on equal footing, and the rest of the README doesn't make much sense to me. [1] and I thought about putting them in my shell project: http://www.oilshell.org/blog/2018/11/30.html http://www.oilshell.org/blog/2018/11/30.html Although QTSV is probably in the more immediate future: https://news.ycombinator.com/item?id=25022836 https://news.ycombinator.com/item?id=25022836
Abishek_Muthian 6y ago
Disabling swap in Linux has helped me in better handling of memory in low memory environments, especially lock-ups even when the code is memory optimised.
The_rationalist 6y ago
I use https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang and it's a game changer
- hakavlad 6y ago
  Look at the new tools: https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd https://github.com/hakavlad/memavaild https://github.com/hakavlad/memavaild It can greatly improve responsiveness. Demo: https://youtu.be/QquulJ06dAo https://youtu.be/QquulJ06dAo - playing supertux + 12 `tail /dev/zero` in background https://youtu.be/DsXEWvq60Rw https://youtu.be/DsXEWvq60Rw - `tail /dev/zero`, swap on HDD, memavaild, no freezes
  - The_rationalist 6y ago
    Very interesting: Should I still use nohang in addition to prelockd and memavaild? I mean prelockd could/should trigger the OOM earlier but nohang would still catch it earlier thanks to PSI? What about zram and zswap advantages? BTW you're helping making the world to be a better place but only nerds download those tools. It would help even more if you could lobby such tools as default in distros such as manjaro, arch, Ubuntu, fedora, etc
Twirrim 6y ago
There's a great talk from a company that had some Netflow SaaS product, that I can't find right now. They were suffering from GC pauses etc. on their ingestion hosts. They spent ages experimenting with various ways of tweaking garbage collection in Java, using different GCs, tweaking settings, etc. They even spent time experimenting with manual memory allocation, but found that to be extremely painful and somewhat fragile. In the end they found that all they really needed to do was just produce less garbage, which was their ultimate "well duh!" revelation. They spent time looking at what was actually producing the garbage, and how they could avoid it. Got rid of a lot of standard coding patterns from their code in favour of new patterns that reduced object allocation, and away went all their problems.

ublaze 6y ago

I've also seen some tricks like stripping docstrings from the release since those strings (hopefully) aren't used in production.

itamarst 6y ago
I can't imagine that has any significant impact on memory usage, at least for normal multi-GB-RAM servers: the docstrings will only be loaded once.
tomwojcik 6y ago
That's nothing. Armin Ronacher was complaining somewhere about the performance hit from static typing in python and i never though about that before. It'd be best if all of that was removed in prod build along with unused imports, kind of what TypeScript does.
- ublaze 6y ago
  imports can't be safely automatically stripped in Python (or even 100% safely re-ordered) since they could have side effects. We run a linter that confirms with the user before stripping them, which works out well in practice.
  - aitchnyu 6y ago
    Its apparently true for RPython too, where side effects which cause "methods and other class attributes do not change after startup".
- aaronbwebber 6y ago
  like PEP484 type annotations having a runtime performance hit? I would be very interested in seeing a link to this if you can dig it up.
- orf 6y ago
  There used to be a performance hit because the annotation objects where all instantiated and attached to the function objects. This is no longer the case - they are only instantiated when inspected.

user5994461 6y ago

Another thing you can do is to deduplicate strings in memory. got 17% memory reduction for loading json objects with some repeated values. https://thehftguy.com/2020/09/25/how-to-deduplicate-string-objects-in-memory-in-python/ https://thehftguy.com/2020/09/25/how-to-deduplicate-string-o...

joshlk 6y ago

DataClassFrames also takes the approach of storing lists of dataclasses as a “dataclass of arrays” - know as data orientated design: https://gitHub.com/joshlk/dataclassframe https://gitHub.com/joshlk/dataclassframe (Full disclosure I’m the author of the project)

chubot 6y ago
FWIW your comment caught my interest as a longstanding user of data frames and Python [1]. But I found the README quite confusing. I guess it's a new project so that's understandable. Why use DataClassFrames and not pandas? Because it's statically typed? The comparison table puts pandas and dataclassframes on equal footing, and the rest of the README doesn't make much sense to me. [1] and I thought about putting them in my shell project: http://www.oilshell.org/blog/2018/11/30.html http://www.oilshell.org/blog/2018/11/30.html Although QTSV is probably in the more immediate future: https://news.ycombinator.com/item?id=25022836 https://news.ycombinator.com/item?id=25022836

Abishek_Muthian 6y ago

Disabling swap in Linux has helped me in better handling of memory in low memory environments, especially lock-ups even when the code is memory optimised.

The_rationalist 6y ago

I use https://github.com/hakavlad/nohang https://github.com/hakavlad/nohang and it's a game changer

hakavlad 6y ago
Look at the new tools: https://github.com/hakavlad/prelockd https://github.com/hakavlad/prelockd https://github.com/hakavlad/memavaild https://github.com/hakavlad/memavaild It can greatly improve responsiveness. Demo: https://youtu.be/QquulJ06dAo https://youtu.be/QquulJ06dAo - playing supertux + 12 `tail /dev/zero` in background https://youtu.be/DsXEWvq60Rw https://youtu.be/DsXEWvq60Rw - `tail /dev/zero`, swap on HDD, memavaild, no freezes
- The_rationalist 6y ago
  Very interesting: Should I still use nohang in addition to prelockd and memavaild? I mean prelockd could/should trigger the OOM earlier but nohang would still catch it earlier thanks to PSI? What about zram and zswap advantages? BTW you're helping making the world to be a better place but only nerds download those tools. It would help even more if you could lobby such tools as default in distros such as manjaro, arch, Ubuntu, fedora, etc

Twirrim 6y ago

There's a great talk from a company that had some Netflow SaaS product, that I can't find right now. They were suffering from GC pauses etc. on their ingestion hosts. They spent ages experimenting with various ways of tweaking garbage collection in Java, using different GCs, tweaking settings, etc. They even spent time experimenting with manual memory allocation, but found that to be extremely painful and somewhat fragile. In the end they found that all they really needed to do was just produce less garbage, which was their ultimate "well duh!" revelation. They spent time looking at what was actually producing the garbage, and how they could avoid it. Got rid of a lot of standard coding patterns from their code in favour of new patterns that reduced object allocation, and away went all their problems.