> The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think

> The "right" way is to build an index of the strings we are storing and then store the index values (hashes of some kind) in the arrays as longs I don't think the rule is that firm: You don't have to use longs because if there are only 256 unique values, why waste so many bytes? Meanwhile if there are so many unique values, is longs enough? > And I am guessing the middleware you want does this transparently? Maybe. I personally think explicit is fine, as long as it's the most easiest and obvious way to do it. But I get that there's a lot of "data scientists" (probably the gross majority) that really struggle with (what I think are) basic data structures -- they'll rarely produce an efficient solution, and we'll see "yet another" article about how awk+sort is faster than a 160 node hadoop cluster... > Maybe storing the index alongside the data in some fashion Maybe not. What works for 30k strings falls over with 300m unique strings. The real trick with nested objects is to invert the query -- to convert the select statements you want to run, into an insert/upsert statement you run when you're loading the data.