3 ms·
I don't think you're contradicting the post. He's talking about the meaning of the concepts and you're talking about how to get efficient implementations. Both
by gruseom 17y ago
I don't think you're contradicting the post. He's talking about the meaning of the concepts and you're talking about how to get efficient implementations. Both are important, but if you didn't have to worry about efficiency, it's pretty obvious that the data/metadata distinction is a logical false turn. (To use your terminology, why wouldn't we want everything to be fast, dynamic, and unified?)
The post does hint at this by suggesting that the concept of metadata itself arose from tradeoffs imposed by hardware. Where I think you differ from Terry is that while he implies that it's time we moved beyond all that, you're saying that the tradeoffs (e.g. between scale and dynamism) remain fundamental. That's a legimitate concern. It's not obvious that we can have our cake and eat it too. It will be interesting to see how FluidDB performs at scale over time.
We've been thinking a lot about these issues on our startup too, because we need both speed and dynamicness. The solution we're arriving at is to exploit the static (structured) cases to get as much speed out of them as we can, be fast-ish on the most common dynamic cases (which thankfully are relatively simple), and be correct-but-slow on the complex dynamic cases. What we won't do, however, unless absolutely forced to, is introduce architectural distinctions between these. We want a consistent, regular system (no two types of data, no two types of anything) that happens to run very fast when handed common usage patterns but still allows for arbitrary flexibility. What we're betting is that the usage patterns will be something like 90% static, 9% dynamic-but-simple, and 1% dynamic-and-complex; we want that to map to 90% super-fast, 9% fast-enough, and 1% slow.
- andreyf 17y agoTo use your terminology, why wouldn't we want everything to be fast, dynamic, and unified? My assumption was that this is provably impossible - that well structured data will always be more efficient to interact with, but on second thought, I can't think of a good reason for that. To the contrary, it seems computational speed is unbounded, while the database size/speed we need are (for most current problems) bounded. Hence, "relatively slow" is going to be "fast enough" more and more often.
- gruseom 17y agoon second thought, I can't think of a good reason for that Isn't it simply that if you know the structure in advance, you don't have to figure out what it is at runtime? I agree that the tradeoff becomes moot if you can get both sides fast enough. The question is whether present-day hardware has gotten us to that point. That's why FluidDB is an interesting test case of whether a system supporting arbitrary flexibility can perform at scale. I assume Terry isn't being naive about this, which means he must be making some tradeoffs somewhere. The question I'd like the answer to is, what battles is he consciously not trying to fight (at a design level)? Terry Jones, if you're reading this, please speak up :)