4 ms·
> wasting up to half the memory OSes generally implement overcommit - allocating a bunch of memory doesn't mean it's actually being used.
by RaisingSpear 3y ago
> wasting up to half the memory
OSes generally implement overcommit - allocating a bunch of memory doesn't mean it's actually being used.
- amluto 3y agoSorry, but allocating approximately 2x the needed size for each string is not going to get covered up well by overcommit unless you are directly mmapping and munmapping (or DONTNEEDing or similar) each allocation and each allocation is more than 8kB. (Probably well over 8kB, and for actual good efficiency in the presence of huge pages, over 4MB so there’s a chance that half of it is unused.) I doubt this workload exists. Overcommit is kind of nifty but not at all magical. I bet a really good AVX-512 transcoder with output size checking could be almost as fast as the OP.
- RaisingSpear 3y ago> unless you are directly mmapping and munmapping (or DONTNEEDing or similar) each allocation You don't free up the memory you're not using? Sounds like you're doing it wrong then. > I bet a really good AVX-512 transcoder with output size checking could be almost as fast as the OP. Submit your pull request and we can evaluate your bet.