2 ms·

> If you want to store numerical keys in a way that will play nicely with sled's iterators and ordered operations, please remember to store your numerical items

> If you want to store numerical keys in a way that will play nicely with sled's iterators and ordered operations, please remember to store your numerical items in big-endian form. Little endian (the default of many things) will often appear to be doing the right thing until you start working with more than 256 items (more than 1 byte), causing lexicographic ordering of the serialized bytes to diverge from the lexicographic ordering of their deserialized numerical form. While I can understand that big endian ordering is convenient for size-indifferent lexical ordering, is the tradeoff worth it? Yes, your benchmarks will look better, but real-world usage will require clients to continually byte swap. So you've basically forced a performance drop into user code. Also, wouldn't size-aware compares perform better in the generated machine code anyway?

rapsey 7y ago
Turning around 8 bytes is a miniscule cost compared to the cost of having a badly formed index. There really is no way around data not being big endian.
- krenoten 7y ago
  yeah, le-be conversions are not generally measurable above noise compared to other database-related work. sled stores arbitrary bytes. endianness is the concern of the person who wants to store higher-level types than bytes, like integers. I do imagine having a story for letting people deserialize/view bytes once, and having that view sit in cache so that hot items are not repeatedly deserialized. I agree with Adya's work " Fast key-value stores: An idea whose time has come and gone " https://research.google/pubs/pub48030/ https://research.google/pubs/pub48030/ where it makes the case that stateful systems should not have to pay repeated deserialization and network costs. I want sled to be well-situated for the world that we're headed into, where this view will become more prominent. This means caching deserialized data, better replication stories, and maybe nice helpers for distributed database authors that allow for atomic tree splits / merges and per-tree replication settings.

rapsey 7y ago

Turning around 8 bytes is a miniscule cost compared to the cost of having a badly formed index. There really is no way around data not being big endian.

krenoten 7y ago
yeah, le-be conversions are not generally measurable above noise compared to other database-related work. sled stores arbitrary bytes. endianness is the concern of the person who wants to store higher-level types than bytes, like integers. I do imagine having a story for letting people deserialize/view bytes once, and having that view sit in cache so that hot items are not repeatedly deserialized. I agree with Adya's work " Fast key-value stores: An idea whose time has come and gone " https://research.google/pubs/pub48030/ https://research.google/pubs/pub48030/ where it makes the case that stateful systems should not have to pay repeated deserialization and network costs. I want sled to be well-situated for the world that we're headed into, where this view will become more prominent. This means caching deserialized data, better replication stories, and maybe nice helpers for distributed database authors that allow for atomic tree splits / merges and per-tree replication settings.