4 ms·
Horrible link title: "They said high level languages relieved us from thinking about memory issues" High level languages do relieve us from thinking about memo
by steventhedev 12y ago
Horrible link title: "They said high level languages relieved us from thinking about memory issues"
High level languages do relieve us from thinking about memory. The linker allows you to forget about program layout in memory and executable. C and other "system" languages free you from managing byte alignment for your structures, byte order and more, and very high level languages even manage your memory use with automatic garbage collection. Locality is a concern in all these languages, and there's no abstraction for that (yet).
The simple solution to the issue presented in the article itself is any of the following:
1. Remove the call to len(). Handle the empty case from the drop/get/peek call, which would appear to be the idiomatic style for Erlang.
2. Track the size of the queue as is suggested in the docs, to improve the efficiency of this specific call.
Or, you could tackle the underlying issue, which is a lack of locality in the list data type following a GC cycle. This would probably involve allocating a chunk of nodes at once so it's the size of a cache line on the platform that's running the VM. Seems to me like that would make a patch worthy of being merged upstream rather than prefetching random chunks of memory.
EDIT: formatting
EDIT: After looking up the offending function: https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_amqqueue_process.erl#L982 https://github.com/rabbitmq/rabbitmq-server/blob/master/src/...
it appears that they can't just remove the call to len, since it is part of the API. So please ignore my first "solution"
- lostcolony 12y agoYeah, RabbitMQ opted for the 2nd because of that. BQ in that case is whatever queue implementation was selected; if you look at both the lqueue.erl and priority_queue.erl, you see that they implemented len() by pulling apart a tuple and taking a value verbatim; no further operations. As such, I'd assume every queue type they support has its own internal wrappers to maintain a length.