6 ms·

> There isn't any point of using C++ to count words in a text file. Any high-level language like Python will beat you to it. However, there's one thing you can

> There isn't any point of using C++ to count words in a text file. Any high-level language like Python will beat you to it. However, there's one thing you can do in C++ and not in Python or JavaScript or PHP: fully control the memory layout of your data. Any books or resources for digging deeper in these kinds of topics? I’m a Python developer and would like to learn C++ but the just the reasons you mentioned make it hard to leave Python.

danudey 6y ago
you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it. You get a pointer to the start of the range, and you know how large it is, so from there it's just pointer arithmetic and assignment. Easy peasy. Now you just have to make sure that you zero data when you're done with it, track how much of that memory is available or used (and which sections), and (optionally) implement your own defragmentation and garbage collection. Unless you desperately need to specifically manage the memory layout of your data (e.g. for alignment purposes for massive data processing), none of this is worth the time or trouble to do for most projects, not only because it's a huge amount of work for little gain, but because you'll almost definitely screw it up and potentially corrupt your entire data set. Note that, at least for the on-disk hashmap set, you absolutely can do this in python, so if you're interested in experimenting with this sort of data management, practice it there first. A good place to start: https://github.com/luispedro/diskhash https://github.com/luispedro/diskhash
- barumi 6y ago
  > you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it. No need for that, as C++ provides placement new(). https://en.cppreference.com/w/cpp/language/new https://en.cppreference.com/w/cpp/language/new
  - saagarjha 6y ago
    This doesn’t really solve the same problem, even though the two often go hand-in-hand.
- gpderetta 6y ago
  It is really not that much work and there are plenty of libs that can do it for you if you do not want to.
john_moscow 6y ago
To get a quick start, try declaring some class variables and then open the Memory window and go to "&variable_name". You can also evaluate "sizeof(variable_name)" to get its exact size. You'll see a bunch of bytes reflecting how exactly your variable is seen by the CPU. That's exactly how many RAM bytes it occupies. It can quickly copied to a new location, limited only by your RAM bandwidth. It can be dumped to a disk and loaded later, limited only by your disk bandwidth.
- saagarjha 6y ago
  How does this help?
nendroid 6y ago
Python developer here just switched jobs to one that does C++. Let me warn you right now. It's not worth it. At all. Don't go down this path. Memory management is easy, but that's not the only crap you have to deal with. Templating and the whole C++ ecosystem with Cmake compiling to make is a giant headache. It takes about a week to integrate a new C++ library in source code and people are worried about installing or "trying" out new libraries because of memory leaks.... while in python I'll install an entire library in seconds just to try out a function. I would say if you made an app in python and the same exact app in C++, the C++ app would perform waay better. No question. But the developer experience and effort required to make that same app is much much more painful.
- rightbyte 6y ago
  It just sounds like you are working on a big messy project.
  - nendroid 6y ago
    Nah it's a big unicorn startup down in Irvine, so the codebase is still relatively new. They use C++17 so pretty modern as well.
- winrid 6y ago
  Why not just fuzz the library if that's a concern?
  - nendroid 6y ago
    The effort required for this makes it not worth it. If I want a library in python it takes about a second to install with one command, and these libraries tend to be safe so you're not going to get any resistance if you want to try out 10 libraries in the next hour or so, which is sort of impossible with c++. Also these libraries are easily uninstalled and isolated from other python apps you have. No need to use massively complex package managers like nix to handle isolation. Fuzzing a library just isn't something that's done in python. The developer just installs it and manually tests it 5 or 6 times in a shell and it's part of the ecosystem in 5 minutes. If you accidentally leave unused libraries in your code and it gets pushed to prod it's a non-issue. I switched to c++ because I was ok with the verbosity and also the complexity and also dealing with memory. But literally every single corner of the c++ ecosystem is a pain in the ass to work with. It's getting to the point where I'm just annoyed all the time. If you want high performance apps go with rust or golang. C++ is just not worth it. For a C++ dev moving to python is like night and day. Imagine a programming environment where you only have to deal with the intention of what your app should do and not all the yak shaving that comes with it on the side. I know some and they tell me they would never go back to c++.
hchz 6y ago
Kill two birds with one stone by learning more about the structure of the Python interpreter and builtins implementations by reading the source.
billfruit 6y ago
Not an exact reply.. but in python dealing with binary data is very inelegant and unintuitive.. Simple things like packing and forming and reading binary packets seems very convoluted in python..

danudey 6y ago

you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it. You get a pointer to the start of the range, and you know how large it is, so from there it's just pointer arithmetic and assignment. Easy peasy. Now you just have to make sure that you zero data when you're done with it, track how much of that memory is available or used (and which sections), and (optionally) implement your own defragmentation and garbage collection. Unless you desperately need to specifically manage the memory layout of your data (e.g. for alignment purposes for massive data processing), none of this is worth the time or trouble to do for most projects, not only because it's a huge amount of work for little gain, but because you'll almost definitely screw it up and potentially corrupt your entire data set. Note that, at least for the on-disk hashmap set, you absolutely can do this in python, so if you're interested in experimenting with this sort of data management, practice it there first. A good place to start: https://github.com/luispedro/diskhash https://github.com/luispedro/diskhash

barumi 6y ago
> you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it. No need for that, as C++ provides placement new(). https://en.cppreference.com/w/cpp/language/new https://en.cppreference.com/w/cpp/language/new
- saagarjha 6y ago
  This doesn’t really solve the same problem, even though the two often go hand-in-hand.
gpderetta 6y ago
It is really not that much work and there are plenty of libs that can do it for you if you do not want to.

john_moscow 6y ago

To get a quick start, try declaring some class variables and then open the Memory window and go to "&variable_name". You can also evaluate "sizeof(variable_name)" to get its exact size. You'll see a bunch of bytes reflecting how exactly your variable is seen by the CPU. That's exactly how many RAM bytes it occupies. It can quickly copied to a new location, limited only by your RAM bandwidth. It can be dumped to a disk and loaded later, limited only by your disk bandwidth.

saagarjha 6y ago
How does this help?

nendroid 6y ago

Python developer here just switched jobs to one that does C++. Let me warn you right now. It's not worth it. At all. Don't go down this path. Memory management is easy, but that's not the only crap you have to deal with. Templating and the whole C++ ecosystem with Cmake compiling to make is a giant headache. It takes about a week to integrate a new C++ library in source code and people are worried about installing or "trying" out new libraries because of memory leaks.... while in python I'll install an entire library in seconds just to try out a function. I would say if you made an app in python and the same exact app in C++, the C++ app would perform waay better. No question. But the developer experience and effort required to make that same app is much much more painful.

rightbyte 6y ago
It just sounds like you are working on a big messy project.
- nendroid 6y ago
  Nah it's a big unicorn startup down in Irvine, so the codebase is still relatively new. They use C++17 so pretty modern as well.
winrid 6y ago
Why not just fuzz the library if that's a concern?
- nendroid 6y ago
  The effort required for this makes it not worth it. If I want a library in python it takes about a second to install with one command, and these libraries tend to be safe so you're not going to get any resistance if you want to try out 10 libraries in the next hour or so, which is sort of impossible with c++. Also these libraries are easily uninstalled and isolated from other python apps you have. No need to use massively complex package managers like nix to handle isolation. Fuzzing a library just isn't something that's done in python. The developer just installs it and manually tests it 5 or 6 times in a shell and it's part of the ecosystem in 5 minutes. If you accidentally leave unused libraries in your code and it gets pushed to prod it's a non-issue. I switched to c++ because I was ok with the verbosity and also the complexity and also dealing with memory. But literally every single corner of the c++ ecosystem is a pain in the ass to work with. It's getting to the point where I'm just annoyed all the time. If you want high performance apps go with rust or golang. C++ is just not worth it. For a C++ dev moving to python is like night and day. Imagine a programming environment where you only have to deal with the intention of what your app should do and not all the yak shaving that comes with it on the side. I know some and they tell me they would never go back to c++.

hchz 6y ago

Kill two birds with one stone by learning more about the structure of the Python interpreter and builtins implementations by reading the source.

billfruit 6y ago

Not an exact reply.. but in python dealing with binary data is very inelegant and unintuitive.. Simple things like packing and forming and reading binary packets seems very convoluted in python..