5 ms·

Hi all, author of Dgraph and the article here. Glad to see this post trending on HN. I'm around if you want to ask any questions, and will share whatever I can

by mrjn 8y ago

Hi all, author of Dgraph and the article here. Glad to see this post trending on HN. I'm around if you want to ask any questions, and will share whatever I can about my experience with graphs and the decisions at Google. Since leaving Google, I've built Dgraph [1]. It is open source, designed for horizontal scalability, ACID transactions and provides incredible performance for both reads and writes. And of course, it solves the join-depth problem as explained in the post. So, do check it out! And do remember to give us a star on GitHub [1]. Your love helps other open source users give Dgraph a real shot in their companies with similar leadership struggles as I experienced at Google. [1]: https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph

Game_Ender 8y ago
I have some experience with Bazel, the open source version of Blaze, and I noticed that it's build graph implementation appears to be very memory hungry as well. Such that a few million lines of code already use 10-30GB of memory. Like you mentioned it requires the entire build graph to fit in memory on one machine. Do you know what sort of graph solution Blaze uses to handle 2 orders of magnitude more code, ie. manage 100's of millions of lines of code? I always assumed it was a distributed graph database, but your article seems to indicate something else.
- mrjn 8y ago
  Google's graph serving system is not generally available to be used by other teams, because its custom-built just for one purpose: to serve knowledge as part of web search infra. I think if Blaze is using some graph solution, it must be custom built.
  - Xorlev 8y ago
    That and the availability of machines with enough RAM to waste.
- shereadsthenews 8y ago
  There are some crumbs that leak out, for example there are mentions of blaze’s “skygraph” feature in the bazel sources.
throwawaygoog10 8y ago
I'm sorry your efforts failed internally. Our infrastructure is somewhat ossified these days: the new and exotic are not well accepted. Other than Spanner (which is still working to replace Bigtable), I can't think of a ton of really novel infrastructure that exists now and didn't when you were around. I mean, we don't even do a lot of generic distributed graph processing anymore. Pregel is dead, long live processing everything on a single task with lots of RAM. I suspect your project would have been really powerful had you gotten the support you needed, but without a (6) or a (7) next to your name it's really hard to convince people of that. I know a number of PAs that would benefit now from structuring their problems in a graph store with arbitrary-depth joins and transactions. I work on one of those and cringe at some of the solutions we've made. We've forgotten what it's like to need novel solutions to performance problems. Instead, we have services we just throw more RAM at. Ganpati is over 128GB (it might even be more) of RAM now, I suspect a solution like dgraph could solve its problems much more efficiently not to mention scalably. Good on you for taking your ideas to market. I'm excited to see how your solution evolves.
- mrjn 8y ago
  One of the TLs was a T7. Got overruled by T8 :-). As Google is growing, there are many T7s around. So, you really need T8 and above to actually get something out of the door, when it comes to impacting web search. P.S. Dgraph is Apache license, written in Go/Protobufs/Grpc. Google might have an appetite for that, worth pushing internally. https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph
  - biomcgary 8y ago
    My understanding is that by default, dgraph query responses are in json. Is it possible to get gRPC responses instead?
    - mrjn 8y ago
      Dgraph exposes both HTTP endpoint and gRPC endpoint. All our official clients use the gRPC endpoint, including Go, Java, NodeJS client. https://docs.dgraph.io/deploy#ports-usage https://docs.dgraph.io/deploy#ports-usage
puzzle 8y ago
Yours is the first public mention of MindMeld I've seen. There were a bunch of other related projects. Some of that experience eventually percolated to Tensorflow. Anyway, good luck with Dgraph! It looks very useful.
v4r 8y ago
What is your opinion on network representation learning? Can it be used in Dgraph? For instance, do you think it is possible to use node embeddings as indices for faster retrieval?
ryanworl 8y ago
To summarize the join-depth problem as you describe, it is because other solutions use hash partitioning instead of range partitioning to distribute data? i.e. if all S/P/O were stored in sorted order such that a given S/P/O were on one machine, you no longer have to do broadcasts and can instead issue queries to just the machines holding that range of the keyspace. Since most "scalable" systems use hash partitioning, they have to broadcast, whereas dgraph uses something more like range partitioning. Or is this too simplistic of an explanation?
- mrjn 8y ago
  It's a good start to solving an array of problems when considering building a low-latency arbitrary depth graph DB. Dgraph does these: 1. It shards by predicate so a full join can be executed by one server. 2. Storing SPO as a record (or row) would still be slow because, in graphs, a single SP can result in thousands or millions of objects. That would involve a lot of iteration and multiple disk seeks to read, which gets slow. So, Dgraph stores them in a posting list format, using SP -> list of O, as a single key-value. 3. That can then result in values which are really large. So, Dgraph converts all nodes into 64-bit integers. 4. Intersections are really important in graphs. So, posting lists store all ints in sorted order. To allow quick intersection between multiple posting lists. 5. Add indices, then replication, HA, transactions, MVCC, correctness, linearizable reads, and ... voila! 3 years later, you have Dgraph! P.S. I intend to find some time and write a paper about the unique design of Dgraph. There's a lot of original research involved in building it.
politician 8y ago
For the past couple of years, I've been working on knowledge graph solutions for the enterprise market, and know acutely the pain of trying to get traction on these types of approaches within one's own organization. Some time before I started working at my current employer, I had a chance to talk to an architect on an EdTech knowledge graph project who advised giving up on these types of efforts due to the complexity of the technology, the politics of the organization, the skills gaps in the engineering corps, and the expectations of customers. So when I ignored that advice and started this project I did have a chance to look at Dgraph, but ultimately avoided it to help relax some of those political and skills constraints. We ended up building a graph serving system out of Cayley processors, but maybe as the skills gap closes from experience on Cayley and the political forces wane as revenues increase then using Dgraph will become viable -- it would help us solve more problems and simplify certain operational headaches. Thanks for writing this experience up. It's encouraging to learn that even giants like Google struggle due to internal factors in this space.
QML 8y ago
I'm a bit late to the thread, but what happened to Cerebro and the concept of a Knowledge Engine? You wrote that it was launched as "Knowledge Strip" but wasn't as impactful as you had intended Cerebro to be.

by mrjn 8y ago

Game_Ender 8y ago

I have some experience with Bazel, the open source version of Blaze, and I noticed that it's build graph implementation appears to be very memory hungry as well. Such that a few million lines of code already use 10-30GB of memory. Like you mentioned it requires the entire build graph to fit in memory on one machine. Do you know what sort of graph solution Blaze uses to handle 2 orders of magnitude more code, ie. manage 100's of millions of lines of code? I always assumed it was a distributed graph database, but your article seems to indicate something else.

mrjn 8y ago
Google's graph serving system is not generally available to be used by other teams, because its custom-built just for one purpose: to serve knowledge as part of web search infra. I think if Blaze is using some graph solution, it must be custom built.
- Xorlev 8y ago
  That and the availability of machines with enough RAM to waste.
shereadsthenews 8y ago
There are some crumbs that leak out, for example there are mentions of blaze’s “skygraph” feature in the bazel sources.

throwawaygoog10 8y ago

I'm sorry your efforts failed internally. Our infrastructure is somewhat ossified these days: the new and exotic are not well accepted. Other than Spanner (which is still working to replace Bigtable), I can't think of a ton of really novel infrastructure that exists now and didn't when you were around. I mean, we don't even do a lot of generic distributed graph processing anymore. Pregel is dead, long live processing everything on a single task with lots of RAM. I suspect your project would have been really powerful had you gotten the support you needed, but without a (6) or a (7) next to your name it's really hard to convince people of that. I know a number of PAs that would benefit now from structuring their problems in a graph store with arbitrary-depth joins and transactions. I work on one of those and cringe at some of the solutions we've made. We've forgotten what it's like to need novel solutions to performance problems. Instead, we have services we just throw more RAM at. Ganpati is over 128GB (it might even be more) of RAM now, I suspect a solution like dgraph could solve its problems much more efficiently not to mention scalably. Good on you for taking your ideas to market. I'm excited to see how your solution evolves.

mrjn 8y ago
One of the TLs was a T7. Got overruled by T8 :-). As Google is growing, there are many T7s around. So, you really need T8 and above to actually get something out of the door, when it comes to impacting web search. P.S. Dgraph is Apache license, written in Go/Protobufs/Grpc. Google might have an appetite for that, worth pushing internally. https://github.com/dgraph-io/dgraph https://github.com/dgraph-io/dgraph
- biomcgary 8y ago
  My understanding is that by default, dgraph query responses are in json. Is it possible to get gRPC responses instead?
  - mrjn 8y ago
    Dgraph exposes both HTTP endpoint and gRPC endpoint. All our official clients use the gRPC endpoint, including Go, Java, NodeJS client. https://docs.dgraph.io/deploy#ports-usage https://docs.dgraph.io/deploy#ports-usage

puzzle 8y ago

Yours is the first public mention of MindMeld I've seen. There were a bunch of other related projects. Some of that experience eventually percolated to Tensorflow. Anyway, good luck with Dgraph! It looks very useful.

v4r 8y ago

What is your opinion on network representation learning? Can it be used in Dgraph? For instance, do you think it is possible to use node embeddings as indices for faster retrieval?

ryanworl 8y ago

To summarize the join-depth problem as you describe, it is because other solutions use hash partitioning instead of range partitioning to distribute data? i.e. if all S/P/O were stored in sorted order such that a given S/P/O were on one machine, you no longer have to do broadcasts and can instead issue queries to just the machines holding that range of the keyspace. Since most "scalable" systems use hash partitioning, they have to broadcast, whereas dgraph uses something more like range partitioning. Or is this too simplistic of an explanation?

mrjn 8y ago
It's a good start to solving an array of problems when considering building a low-latency arbitrary depth graph DB. Dgraph does these: 1. It shards by predicate so a full join can be executed by one server. 2. Storing SPO as a record (or row) would still be slow because, in graphs, a single SP can result in thousands or millions of objects. That would involve a lot of iteration and multiple disk seeks to read, which gets slow. So, Dgraph stores them in a posting list format, using SP -> list of O, as a single key-value. 3. That can then result in values which are really large. So, Dgraph converts all nodes into 64-bit integers. 4. Intersections are really important in graphs. So, posting lists store all ints in sorted order. To allow quick intersection between multiple posting lists. 5. Add indices, then replication, HA, transactions, MVCC, correctness, linearizable reads, and ... voila! 3 years later, you have Dgraph! P.S. I intend to find some time and write a paper about the unique design of Dgraph. There's a lot of original research involved in building it.

politician 8y ago

For the past couple of years, I've been working on knowledge graph solutions for the enterprise market, and know acutely the pain of trying to get traction on these types of approaches within one's own organization. Some time before I started working at my current employer, I had a chance to talk to an architect on an EdTech knowledge graph project who advised giving up on these types of efforts due to the complexity of the technology, the politics of the organization, the skills gaps in the engineering corps, and the expectations of customers. So when I ignored that advice and started this project I did have a chance to look at Dgraph, but ultimately avoided it to help relax some of those political and skills constraints. We ended up building a graph serving system out of Cayley processors, but maybe as the skills gap closes from experience on Cayley and the political forces wane as revenues increase then using Dgraph will become viable -- it would help us solve more problems and simplify certain operational headaches. Thanks for writing this experience up. It's encouraging to learn that even giants like Google struggle due to internal factors in this space.

QML 8y ago

I'm a bit late to the thread, but what happened to Cerebro and the concept of a Knowledge Engine? You wrote that it was launched as "Knowledge Strip" but wasn't as impactful as you had intended Cerebro to be.