3 ms·

What are you thinking about for the API signatures? Without beating a dead horse this is where go’s lack of generics really becomes burdensome. If it were an o

What are you thinking about for the API signatures? Without beating a dead horse this is where go’s lack of generics really becomes burdensome. If it were an on disk cache []byte would likely make sense but given it’s in memory I’m not sure. If you use interface{} you’ll need to measure that cost as well.

amanmangal 8y ago
Our current plan is to store []byte given that badger stores blobs of bytes. Using interface{} may have some overhead, but we are no so worried about that right now.
- kasey_junk 8y ago
  So the []byte solution requires running through a serielization step which for most in memory uses will be expensive. Did you choose that because in you use case it eventually ends up setialized in any case?
  - amanmangal 8y ago
    correct!
np_tedious 8y ago
I'm a go novice, and don't have any particular knowledge about dgraph, but is this a case where auto-generated code could be a good solution? (Assuming it's a bounded set of types)
Thaxll 8y ago
I don't see why it's a problem, caching solutions like Redis also store bytes. If you cache a JSON blob you don't have much choice.
- kasey_junk 8y ago
  For an in memory cache (ie one not going to disk/network) you have to unnecessarily serialize your in memory representation to bytes. This is very frequently the most expensive (computational) step in a system. If you are just going to send that stuff to nic/disk eventually anyway its perfectly fine (which is why something like Redis which is an out of memory cache can get away with it).
mrjn 8y ago
As mentioned down in this thread, the key-val would be bytes-bytes or string-bytes. While this is more expensive in terms of CPU, it has the advantage that the size of elements in the cache are carefully understood and do not need the cost of GC sweeps. When de-serialized, in-memory structs take up more memory on heap and it gets hard to compute that -- we've tried that in vain in the past. Having said that, there's still a general need to store pointers as values and that's something we'll look at once this byte-based implementation is ready.

amanmangal 8y ago

Our current plan is to store []byte given that badger stores blobs of bytes. Using interface{} may have some overhead, but we are no so worried about that right now.

kasey_junk 8y ago
So the []byte solution requires running through a serielization step which for most in memory uses will be expensive. Did you choose that because in you use case it eventually ends up setialized in any case?
- amanmangal 8y ago
  correct!

np_tedious 8y ago

I'm a go novice, and don't have any particular knowledge about dgraph, but is this a case where auto-generated code could be a good solution? (Assuming it's a bounded set of types)

Thaxll 8y ago

I don't see why it's a problem, caching solutions like Redis also store bytes. If you cache a JSON blob you don't have much choice.

kasey_junk 8y ago
For an in memory cache (ie one not going to disk/network) you have to unnecessarily serialize your in memory representation to bytes. This is very frequently the most expensive (computational) step in a system. If you are just going to send that stuff to nic/disk eventually anyway its perfectly fine (which is why something like Redis which is an out of memory cache can get away with it).

mrjn 8y ago

As mentioned down in this thread, the key-val would be bytes-bytes or string-bytes. While this is more expensive in terms of CPU, it has the advantage that the size of elements in the cache are carefully understood and do not need the cost of GC sweeps. When de-serialized, in-memory structs take up more memory on heap and it gets hard to compute that -- we've tried that in vain in the past. Having said that, there's still a general need to store pointers as values and that's something we'll look at once this byte-based implementation is ready.