2 ms·
Not only that, but the author takes a single data point and attempts to draw a line. Without measuring 10k, 20k, 30k of each and seeing how the memory usage ch
by coder543 6y ago
Not only that, but the author takes a single data point and attempts to draw a line.
Without measuring 10k, 20k, 30k of each and seeing how the memory usage changes, we are seriously comparing apples to oranges here.
Go's allocator will hold onto a decent chunk of memory just to amortize the cost of reaching out to the operating system. Just asking the operating system how much memory the Go application is using doesn't tell the whole story in this benchmark.
Is it 3x or 30x? The author certainly doesn't know based on taking a single data point from each application.
Goroutines are more memory efficient than full OS threads, but they're also cheaper to spawn and the Go runtime can optimize for certain common use cases very effectively.
I've also personally had serious difficulty in the past with getting Linux distros to let me launch tens of thousands of OS threads. Long before I run out of memory, I hit various limits that block the program from spawning additional threads. Some of them can be adjusted, but I never managed to spawn threads arbitrarily up to the amount of memory that was available... probably just my own failing, but that alone is reason enough not to spawn an unbounded number of OS threads.
- labawi 6y agoThere was a related post about a month ago[1]. I believe the article fails to account for, among other things, the virtual memory overhead that comes with sparse allocations for large maximum stack sizes, that would about double the numbers. With 2MB or so stack spacing, if you only use a single page of RAM per thread, you still need another whole page of RAM in the "page table" (actually a trie) and on linux those are not counted in the process RES memory usage. That being said, creating thousands of native threads has lots of other pitfalls, including stack ones - would not recommend as a general strategy. If haven't tried creating actual threads, but for test mapping stack-like staggered memory, I've had success with: sysctl vm.overcommit_memory=1 sysctl vm.max_map_count=10000000 swapoff -a Remember to keep an eye on free memory, not process RES or similar, that doesn't count page tables and other overhead. [1] https://news.ycombinator.com/item?id=25997506 https://news.ycombinator.com/item?id=25997506
- matklad 6y agoThanks, RSS not counting memory for page tables themselves is a good point, I will correct that in the article tomorrow.
- labawi 6y agoNote that page tables are the per-process virtual memory mapping data structure for the CPU. AFAICT, Linux has separate accounting of the memory, which is smaller, but also per-thread structures, which are likely not included in RSS either.