4 ms·

We did measure significantly bigger memory usage with IndexMap and needed to revert back to HashMap eventually. Deserializing into a defined struct does not wa

by pimeys 2y ago

We did measure significantly bigger memory usage with IndexMap and needed to revert back to HashMap eventually. Deserializing into a defined struct does not waste as much memory as Value does. Especially due to the recursive nature of the Map variant, which can hold another Map.

masklinn 2y ago
> We did measure significantly bigger memory usage with IndexMap and needed to revert back to HashMap eventually. That is strange and I’d assume the maintainers would be interested in the information. By my reckoning HashMap would be consuming about capacity * 10/9 * (8 + sizeof key + sizeof value) while indexmap should be consuming capacity * 10/9 * 8 + capacity * (8 + sizeof key + sizeof value). Unless indexmap reuses hashbrown directly in which case you’d get something like capacity * 10/9 (16 + sizeof key) + capacity * sizeof value.
- SkiFire13 2y ago
  Indexmap does reuse hashbrown. It consists of a hashtable containing `usize` indexes into a `Vec` which in turn contains the actual entries (keys and values), along with a cached hash for the key. In the end the overhead should only be that index and the hash.
- pimeys 2y ago
  It's pretty easy to try out: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=6bfcdc0a94fcbb45cc430ab139a962cb https://play.rust-lang.org/?version=stable&mode=debug&editio... Now, if you have an enum such as `serde_json::Value` which is kind of recursive with its `Map` variant, and you have a ton of dynamic JSON parsing in your code, these numbers really add up. And serde_json uses `BTreeMap` (I was wrong in my previous message) by default which is even smaller than `HashMap`. The learning here is to avoid dynamic JSON parsing if you can. And if needing it, but not caring the insertion order, avoid the `preserve_order` feature flag. The other learning here is that there is no map structure that fits to all purposes. They all have their pros and cons and you should choose the right one for the problem.

by pimeys 2y ago

masklinn 2y ago

> We did measure significantly bigger memory usage with IndexMap and needed to revert back to HashMap eventually. That is strange and I’d assume the maintainers would be interested in the information. By my reckoning HashMap would be consuming about capacity * 10/9 * (8 + sizeof key + sizeof value) while indexmap should be consuming capacity * 10/9 * 8 + capacity * (8 + sizeof key + sizeof value). Unless indexmap reuses hashbrown directly in which case you’d get something like capacity * 10/9 (16 + sizeof key) + capacity * sizeof value.

SkiFire13 2y ago
Indexmap does reuse hashbrown. It consists of a hashtable containing `usize` indexes into a `Vec` which in turn contains the actual entries (keys and values), along with a cached hash for the key. In the end the overhead should only be that index and the hash.
pimeys 2y ago
It's pretty easy to try out: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=6bfcdc0a94fcbb45cc430ab139a962cb https://play.rust-lang.org/?version=stable&mode=debug&editio... Now, if you have an enum such as `serde_json::Value` which is kind of recursive with its `Map` variant, and you have a ton of dynamic JSON parsing in your code, these numbers really add up. And serde_json uses `BTreeMap` (I was wrong in my previous message) by default which is even smaller than `HashMap`. The learning here is to avoid dynamic JSON parsing if you can. And if needing it, but not caring the insertion order, avoid the `preserve_order` feature flag. The other learning here is that there is no map structure that fits to all purposes. They all have their pros and cons and you should choose the right one for the problem.