6 ms·
Beam: Distributed knowledge graph store written in Go by eBay
- zenlot 7y agoI wonder if name is coincidence or it's on purpose to get some traffic from searching on Erlang BEAM. https://en.m.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine) https://en.m.wikipedia.org/wiki/BEAM_(Erlang_virtual_machine...
- diehunde 7y agoI was wondering the same but for Apache Beam. I don't think it's to get traffic. Just coincidence.
- OrgNet 7y ago... or maybe they chose an English word to make it harder to find.
- OrgNet 7y agousing two English words as the name would already be an improvement for the common web-searcher... As an example, here is the name of two apps that do similar things: "Blue iris" and "motion".... let me know which one that you are able find more quickly.
- umhummsmu 7y agoParanoid much?
- isuckatcoding 7y agoAh yes but not to be confused with apache beam.
- alexchamberlain 7y agoThis is very interesting. The triple store space is crying out for a good solution; a good _distributed_ solution - unheard of.
- flarg 7y agoIs it common to combine triple store with event logging as this seems to do?
- alexchamberlain 7y agoI may have misunderstood, but I think they're using the log as a write ahead log, which is a similar, but subtly different idea to event logging. A Write Ahead Log is an implementation detail of a distributed system, whereby the system has a log of actions it applies against a state machine. Protocols, such as Raft, exist to keep the log in sync, and the state machine is kept in sync as a consequence. The log entries would be quite abstract and low-level, like "tuple inserted". An Event Log, on the other hand, is an application level architecture technique. The log entries would be application-specific actions, like "user logged in".
- frou_dh 7y agoOBSERVATION: 7 out of 9 comments are bikeshedding the project's name.
- bruxis 7y agoAnd if we include yours (and mine), we're at 9 out of 11!
- fulafel 7y ago"Beam is designed to store large graphs that cannot fit on a single server. " What would be a good alternative for less huge datasets?
- manigandham 7y agoA single-node graph database? Neo4j is probably the most well-supported and standardized option: https://neo4j.com/ https://neo4j.com/
- snaky 7y agoAllegroGraph https://franz.com/agraph/allegrograph/ https://franz.com/agraph/allegrograph/
- hu3 7y agohttps://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph is worth a look. Although it can be distributed, setting it up is trivial enough to be appealing in single machine configurations.
- fouc 7y agoI'm curious how Erlang's BEAM/OTP would handle storing large graphs across servers. (My understanding is that it can probably support that?)
- RobertKerans 7y agoOut of the box, imo it'll be faffy and probably not worth it vs just using an existing solution. :digraph is great, having a graph structure available out of the box in the standard library is super nice for certain things, but it's 3 ETS tables, and ETS tables can't be replicated across nodes. MNesia tables can (that's the point), so replicating how :digraph works using MNesia is probably fairly doable (I assume this has been tried but I haven't investigated it)
- dvirsky 7y agoThe fastest one would probably be RedisGraph. https://oss.redislabs.com/redisgraph/ https://oss.redislabs.com/redisgraph/
- deleted 7y ago[deleted]
- eternalban 7y agoArchitectural decisions explained here: https://github.com/eBay/beam/blob/master/docs/central_log_arch.md https://github.com/eBay/beam/blob/master/docs/central_log_ar... The architecture is: materialized views from a central append log. I don't see a reference to Apache Bookkeeper[1][2] in the above. My recommendation to anyone who wants to build their own variant of this architecture is to build it on top of Bookkeeper instead of rolling your own. It is designed for this precise workload [2] and addresses the write throughput limitations of the OP 'Beam'. [edit: pull quote from [2]]: "what makes BookKeeper unique is its ability to offer a short-tailed, low-latency, distributed scale-out storage solution. Although this is a CP system, its greater availability makes it almost a C(A)P system. It is an apt storage for immutable data." [1]: https://bookkeeper.apache.org/ https://bookkeeper.apache.org/ [2]:https://www.linux.com/news/event/apachecon/2017/4/high-performance-logging-apache-bookkeeper https://www.linux.com/news/event/apachecon/2017/4/high-perfo...
- refset 7y ago"some functionality is lacking before Beam could be used as a critical production data store, including deletion of facts" It sounds like this needs a second central log to enable eviction of sensitive data before it can be seriously considered usable. One log for transactions and another other for the data itself. Or at least that is the conclusion we reached when designing and building our own knowledge graph store (with a similar use of Kafka): https://juxt.pro/crux/docs/index.html#_unbundled https://juxt.pro/crux/docs/index.html#_unbundled
- b_tterc_p 7y agowhat is the best general use case open source knowledge graph, if any? The actual graph, not the tech.