2 ms·
Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you) As far as
by lifeisstillgood 9y ago
Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you)
As far as I understand you want to handle nested arrays of strings in your data. Ok
The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs
This way our arrays are doing numbers and we handwavy search for or use strings through some wrapper
Is this right?
And I am guessing the middleware you want does this transparently? Maybe storing the index alongside the data in some fashion
- geocar 9y ago> The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.