4 ms·
When I was building an object store years ago; I needed a way to attach metadata tags to each object. The objects themselves could be files like a picture, a do
by didgetmaster 2y ago
When I was building an object store years ago; I needed a way to attach metadata tags to each object. The objects themselves could be files like a picture, a document, or some music; and I wanted to allow tags to denote things like the author, the camera, or the music genre.
Most systems use things like file extended attributes or a separate database to store such metadata; but I wanted something different. It needed to be able to attach tags to hundreds of millions of objects and find things that matched certain tags quickly.
I invented a key-value store to hold the metadata and got it working well. When it started to look like a big columnar store with sparsely populated rows; I decided to see if it could handle queries like a relational database. To my surprise it not only did it well, it could outperform many of them.
There are data models besides relational that can work extremely well for certain data sets.
- whartung 2y agoI'm not going to have 100's of millions of objects, so I can't speak to that. But for my one hobby project, I'm using RDF and a triple store. Even with a "small" dataset, you can get an explosion of properties. I want to be able to add arbitrary properties to arbitrary things and relate them all together. Build the graph organically. So far, its working really well. But underneath, its (likely) just a couple of b+trees do all of the heavy lifting.
- enord 2y agoMost vendors use three indexes for triples and 4 or 6 for quads. All the indexes are covering, which is to say they triplicate all data—-in other words the database consists only of indexes. Aint that just neat?
- whartung 2y agoI refer to RDF as the "absurd normal form". When my friends and I do DB design, it's almost inevitable we fall into what might inevitably become the "ThingThing" table thats a many-to-many joiner of everything to everything else. (That's when we giggle, leave the room, go to lunch, and then come back when we've returned to our senses.) But, for RDF its exactly what I want, I'm not interested in schemas and such for this work, so it's perfect for my scenario.
- enord 2y agoThe killer feature isn’t even fully indexed queries for ever, it’s the serialization formats. Need to do a non-trivial merge of complex domain graphs? Why have you tried string concatenating turtle files?