4 ms·

Hi, thank you for the really cool post! I am trying to understand some key concepts here, so please forgive the simple questions: > 16byte subject/entity | 16

Hi, thank you for the really cool post! I am trying to understand some key concepts here, so please forgive the simple questions: > 16byte subject/entity | 16 byte predicate/attribute | 32 byte object/value What would be a difference between subject and entity? Do you include a timestamp of your entries next to your trible? > Having a fixed size binary representation makes this compatible with most existing databases (...) Are you using a external look up table to identify the human language definition of the entry, and keep using the 2^128 possible entries for internal use? > (...) if you want to upgrade your schema (...) > We stole the context from jsonLD (...) What were the reasons you did not use jsonLD as a base for your software? Could you point perhaps point me to a case study of your system, or, if this is not possible, a similar case published in literature/www etc? I would love to learn more what you are doing (my contact is in the my profile). Wish I could upvote you a couple of times. Thank you.

j-pb 6y ago
Glad that you like it :D This actually pushes me a bit more into the direction of open-sourcing the whole thing, we kinda have it planned, but it's not a priority at the moment, because we use it ourselves quite happily :D. Subject and Entity are the same thing, just different names for it. People with a Graph DB background will more commonly use [entity attribute value] for triples, while people from the Semantic Web community, commonly use [subject predicate object]. We don't use timestamps, but we just implemented something we call UFO-IDs (Unique, Forgettable, Ordered), where we store a 1 second resolution timer in the first 16 bit, which improves data locality and allows us to forget irrelevant tribles within a 18h window (which is pretty nice if you do e.g. robotics or virtual personal assistants), while at the same time still practicing the overflow case regularly (in comparison to UUIDv1, ULID, or Timeflakes), and not loosing too many bits of entropy (especially in cases where the system runs longer than 18h). The 128bit is actually big enough though that you can just choose any random value, and be pretty darn certain that it's unique. (UUIDv4 works that way) 64 byte / 512bit not only fits into cache lines, it's also the smallest value, which is statistically "good enough". 128 bit random IDs (entity and attribute) are statistically unlikely enough to collide, and 256bit hashes (the value) are likewise good enough for the foreseeable future to avoid content collision. And yeah well, the human language name, as well as all the documentation about the attribute is actually stored as tribles alongside the data. We use them for code generation for statically typed programming languages which allows us to hook into the languages type checker, to create documentation on the fly, and to power a small ontology editing environment (take that protege ;) ). We kinda use it as a middleware, similar to ROS, so it has to fit into the same soft realtime, static typing, compile everything nich, while at the same time allowing for explorative programming in dynamic languages like Javascript. We use observableHQ notebooks to do all kinds of data-analysis, so naturally we want a nice workflow there. jsonLD is heavily hooked into the RDF ecosystem. We actually started in the RDF space, but it became quickly apparent that the overall complexity was a show stopper. Originally this was planned to bring the sub projects in a large EU research project closer together, and encourage collaboration. We found that every sub project wanted to be the Hub that connected all the other ones. By having a 2.5kloc implementation we figured, everybody could "own" the codebase, and associate with it, make it so stupid and obvious that everybody feels like they came up with the idea themselves. The good old inverse Conway manoeuvre. jsonLD is also very static, in the ways that it allows you to reinterpret data, RDF in =churn=> JSON out, and we wanted to be able to do so dynamically, so that when you refactor code to use new attribute variants (e.g. with different deserialisations) you can do so gradually. Also dynamic is a lot faster. The tribles instead of triples idea came when we've noticed that basically every triple store implementation does a preprocessing step where CURI are converted to u64 / 8byte integer to be stored in the indices. We just went: "Well, we could either put 24 byte in the index and still have to do 3 additional lookups. Or we could put 64 byte (2.5x) in there and get range queries, sorting, and no additional lookups, with essentially the same write and read characteristics.[Because our Adaptive Radix Tree index compresses all the random bits.]" 64 bit words are already pretty darn big... Currently there is nothing published (except for it being vaguely mentioned in some linguistics papers), and no studies done. They are planned though, but as this isn't our source of income it's lowish priority (much to my dismay :D). Keep an eye on tribles.space tho ;) Edit: Ah well, why wait, might as well start building a community :D https://discord.gg/KP5HBYfqUf https://discord.gg/KP5HBYfqUf

j-pb 6y ago

Glad that you like it :D This actually pushes me a bit more into the direction of open-sourcing the whole thing, we kinda have it planned, but it's not a priority at the moment, because we use it ourselves quite happily :D. Subject and Entity are the same thing, just different names for it. People with a Graph DB background will more commonly use [entity attribute value] for triples, while people from the Semantic Web community, commonly use [subject predicate object]. We don't use timestamps, but we just implemented something we call UFO-IDs (Unique, Forgettable, Ordered), where we store a 1 second resolution timer in the first 16 bit, which improves data locality and allows us to forget irrelevant tribles within a 18h window (which is pretty nice if you do e.g. robotics or virtual personal assistants), while at the same time still practicing the overflow case regularly (in comparison to UUIDv1, ULID, or Timeflakes), and not loosing too many bits of entropy (especially in cases where the system runs longer than 18h). The 128bit is actually big enough though that you can just choose any random value, and be pretty darn certain that it's unique. (UUIDv4 works that way) 64 byte / 512bit not only fits into cache lines, it's also the smallest value, which is statistically "good enough". 128 bit random IDs (entity and attribute) are statistically unlikely enough to collide, and 256bit hashes (the value) are likewise good enough for the foreseeable future to avoid content collision. And yeah well, the human language name, as well as all the documentation about the attribute is actually stored as tribles alongside the data. We use them for code generation for statically typed programming languages which allows us to hook into the languages type checker, to create documentation on the fly, and to power a small ontology editing environment (take that protege ;) ). We kinda use it as a middleware, similar to ROS, so it has to fit into the same soft realtime, static typing, compile everything nich, while at the same time allowing for explorative programming in dynamic languages like Javascript. We use observableHQ notebooks to do all kinds of data-analysis, so naturally we want a nice workflow there. jsonLD is heavily hooked into the RDF ecosystem. We actually started in the RDF space, but it became quickly apparent that the overall complexity was a show stopper. Originally this was planned to bring the sub projects in a large EU research project closer together, and encourage collaboration. We found that every sub project wanted to be the Hub that connected all the other ones. By having a 2.5kloc implementation we figured, everybody could "own" the codebase, and associate with it, make it so stupid and obvious that everybody feels like they came up with the idea themselves. The good old inverse Conway manoeuvre. jsonLD is also very static, in the ways that it allows you to reinterpret data, RDF in =churn=> JSON out, and we wanted to be able to do so dynamically, so that when you refactor code to use new attribute variants (e.g. with different deserialisations) you can do so gradually. Also dynamic is a lot faster. The tribles instead of triples idea came when we've noticed that basically every triple store implementation does a preprocessing step where CURI are converted to u64 / 8byte integer to be stored in the indices. We just went: "Well, we could either put 24 byte in the index and still have to do 3 additional lookups. Or we could put 64 byte (2.5x) in there and get range queries, sorting, and no additional lookups, with essentially the same write and read characteristics.[Because our Adaptive Radix Tree index compresses all the random bits.]" 64 bit words are already pretty darn big... Currently there is nothing published (except for it being vaguely mentioned in some linguistics papers), and no studies done. They are planned though, but as this isn't our source of income it's lowish priority (much to my dismay :D). Keep an eye on tribles.space tho ;) Edit: Ah well, why wait, might as well start building a community :D https://discord.gg/KP5HBYfqUf https://discord.gg/KP5HBYfqUf