4 ms·

A new kind of Key/Value store. In this architecture, keys and values are stored within two separate 'data objects' which are linked together. One holds all the

A new kind of Key/Value store. In this architecture, keys and values are stored within two separate 'data objects' which are linked together. One holds all the unique values along with their reference counts, the other contains all the keys and links to their mapped values. The architecture allows any value to be mapped to one or more keys and any key to be mapped to one or more values (unless an attribute on the data object prevents it). These KV stores can be used to attach tags to other objects. They can be used to form columns in relational tables (i.e. a columnar store). It can be used to create indexes into file contents. Each object is designed for parallel access by multiple threads. Only a single block needs to be locked to update any value or key so multiple writes can occur at the same time. The data in each object is organized to find values or keys without inspecting every block. It is very fast and allows both OLTP and OLAP operations on the same data set. So far, I have used them to attach tags to millions of files and do searches. I have created relational tables that I can query much faster than the same table in Postgres (https://www.youtube.com/watch?v=OVICKCkWMZE https://www.youtube.com/watch?v=OVICKCkWMZE). I have created 3D relational tables that can be queried the same way 2D tables have traditionally been queried (https://www.youtube.com/watch?v=AXqvWMmoL1M https://www.youtube.com/watch?v=AXqvWMmoL1M). The software is currently in an open beta and anyone can download and try it out on their own data set. https://www.Didgets.com https://www.Didgets.com

rileyphone 4y ago
Sounds kinda like tuplespaces/Linda (which I think are very good, largely unpursued ideas).
- didgetmaster 4y ago
  It might have some things in common. In my case, the objects are persistent and can be read to and written from disk with ease. The relationship between keys and values can be 1:1, 1:many, many:1, or many:many. If the KV store is a 'state' column in a DB table of US customer addresses for example, the number of unique values is limited to 50 (if you ignore D.C., Puerto Rico, etc.). The table might have 100M customers, each mapped to at least one state. Each state (except less populated ones like Wyoming) might have millions of rows mapped to it. Many customers might have addresses in more than one state. The system has to do things like: "Find every key mapped to a state that starts with the letter 'M'" or "Find every value that is mapped to this set of keys". These are the kinds of things a normal SQL query has to do.
pphysch 4y ago
What's the practical value of (quickly) knowing which keys share the same value? "pet":"cat" and "program":"cat" have different semantics.
- didgetmaster 4y ago
  The 'key' in this case is what the tag or value is attached to, not the context of the value. So in your example, all the "pet" values would be stored together within the KV store just like values in a relational table column. Likewise all the "program" values would be stored together within a different KV store. So some typical queries might be: Find all the photos where "Event" = "Wedding" Find all rows where "State" = "California" The keys in this case would be the IDs of the photos or the row keys in the relational table.

rileyphone 4y ago

Sounds kinda like tuplespaces/Linda (which I think are very good, largely unpursued ideas).

didgetmaster 4y ago
It might have some things in common. In my case, the objects are persistent and can be read to and written from disk with ease. The relationship between keys and values can be 1:1, 1:many, many:1, or many:many. If the KV store is a 'state' column in a DB table of US customer addresses for example, the number of unique values is limited to 50 (if you ignore D.C., Puerto Rico, etc.). The table might have 100M customers, each mapped to at least one state. Each state (except less populated ones like Wyoming) might have millions of rows mapped to it. Many customers might have addresses in more than one state. The system has to do things like: "Find every key mapped to a state that starts with the letter 'M'" or "Find every value that is mapped to this set of keys". These are the kinds of things a normal SQL query has to do.

pphysch 4y ago

What's the practical value of (quickly) knowing which keys share the same value? "pet":"cat" and "program":"cat" have different semantics.

didgetmaster 4y ago
The 'key' in this case is what the tag or value is attached to, not the context of the value. So in your example, all the "pet" values would be stored together within the KV store just like values in a relational table column. Likewise all the "program" values would be stored together within a different KV store. So some typical queries might be: Find all the photos where "Event" = "Wedding" Find all rows where "State" = "California" The keys in this case would be the IDs of the photos or the row keys in the relational table.