3 ms·

Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran int

by mwambua 10y ago

Very interesting. I think the reality in a lot of situations is that most people don't really need the full feature-set that graph databases provide. I ran into a similar problem trying to explore Wikidata's json dumps. It's a lot simpler to load it into MongoDB and create indices where you need them, rather than figuring out how to interface to a proprietary system that you may or may not end up using in the long run. I'm still having trouble keeping my indices in memory though, and would be keen to know what sort of latency you encounter hitting an on-disk hashtable.

rspeer 10y ago
What I end up with is a 4 GB index, whose contents are byte offsets into an 8 GB file containing the properties of the edges. When I mmap these, an in-memory lookup takes about 1 ms, but I can have unfortunate delays of like 100 ms per lookup if I'm starting cold or hitting things that are swapped out of memory.
rspeer 10y ago
Also, yeah, I don't really need most of the things that graph DBs are offering. They seem to focus a lot on making computationally unreasonable things possible -- such as arbitrary SPARQL queries. I'm not the kind of madman who wants to do arbitrary SPARQL queries. I just want to be able to load data quickly, look up the edges around a node quickly, and then once I can do that, I'd also like to be able to run the occasional algorithm that propagates across the entire graph, like finding the degree-k core.
Tpt 10y ago
I use MongoDB for a Wikidata replica and index performances are quite good. I use some hacks in order to keep size of indexed values low (see https://github.com/ProjetPP/WikibaseEntityStore/blob/master/src/MongoDB/MongoDBDocumentBuilder.php#L86 https://github.com/ProjetPP/WikibaseEntityStore/blob/master/... ). It helps a lot in order to be able to keep indexes into memory. It powers https://askplatyp.us https://askplatyp.us quite well.

by mwambua 10y ago

rspeer 10y ago

What I end up with is a 4 GB index, whose contents are byte offsets into an 8 GB file containing the properties of the edges. When I mmap these, an in-memory lookup takes about 1 ms, but I can have unfortunate delays of like 100 ms per lookup if I'm starting cold or hitting things that are swapped out of memory.

rspeer 10y ago

Also, yeah, I don't really need most of the things that graph DBs are offering. They seem to focus a lot on making computationally unreasonable things possible -- such as arbitrary SPARQL queries. I'm not the kind of madman who wants to do arbitrary SPARQL queries. I just want to be able to load data quickly, look up the edges around a node quickly, and then once I can do that, I'd also like to be able to run the occasional algorithm that propagates across the entire graph, like finding the degree-k core.

Tpt 10y ago

I use MongoDB for a Wikidata replica and index performances are quite good. I use some hacks in order to keep size of indexed values low (see https://github.com/ProjetPP/WikibaseEntityStore/blob/master/src/MongoDB/MongoDBDocumentBuilder.php#L86 https://github.com/ProjetPP/WikibaseEntityStore/blob/master/... ). It helps a lot in order to be able to keep indexes into memory. It powers https://askplatyp.us https://askplatyp.us quite well.