3 ms·

I often want one of these middlewares. Strings are a killer -- indeed any variable-length object makes array programming tricky when it's nested, so a sound st

by geocar 9y ago

I often want one of these middlewares. Strings are a killer -- indeed any variable-length object makes array programming tricky when it's nested, so a sound strategy is to intern your strings first (in some way; KDB has enumerations, but language support isn't necessary: hash the strings and save an inverted index works good enough for a lot of applications). Interning strings means you see integers in your data operations, which is about as un-fun to program in as it sounds. People want to be able to write something like: ….str.extract('([ab])(\d)', expand=False) and then get disappointed that it's slow. Everything is slow when you do it a few trillion times, but slow things are really slow when you do them a few trillion times. If we think about how we build our tables, we can store these as a single-byte column (or even a bitmask) and an int (or long) column, then we get fast again. However it is clear "fast" and "let's use JSON" are incompatible, and a good middleware or storage system isn't going to make me trade.

lifeisstillgood 9y ago
Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you) As far as I understand you want to handle nested arrays of strings in your data. Ok The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs This way our arrays are doing numbers and we handwavy search for or use strings through some wrapper Is this right? And I am guessing the middleware you want does this transparently? Maybe storing the index alongside the data in some fashion
- geocar 9y ago
  > The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.

by geocar 9y ago

lifeisstillgood 9y ago

Could you expand on this (I'm asking because you clearly intend something you have thought about a lot, but I am missing the point - it's me not you) As far as I understand you want to handle nested arrays of strings in your data. Ok The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs This way our arrays are doing numbers and we handwavy search for or use strings through some wrapper Is this right? And I am guessing the middleware you want does this transparently? Maybe storing the index alongside the data in some fashion

geocar 9y ago
> The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.