5 ms·

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check

by mrjn 6y ago

Hey folks, author of Dgraph here. If you're interested in the design details of Dgraph, and have the appetite for a very technical research paper, please check this out: https://dgraph.io/paper https://dgraph.io/paper I'd like to thank Kyle in doing another round of testing. Some of these bugs that we fixed (in 2018 and 2019) were very tricky edge cases -- it's incredible to see Dgraph running so much more stable now, under all varied failure scenarios. Let me know if you have any questions, I'm around to answer.

bullen 6y ago
Did you have to pay Kyle for him to test your database?
- alexchamberlain 6y ago
  The report states the work was funded by Dgraph and conducted in line with their ethics policy.
- sorokod 6y ago
  You may find the Ethics [1] section useful. Also, are you an acquaintance of Kyle Kingsbury? Just curious as you refer to him by his first name. [1] https://jepsen.io/ethics https://jepsen.io/ethics
  - dang 6y ago
    It's an internet convention to refer to people by their first names.
- aphyr 6y ago
  Yep! Testing databases is my full time job. Vendors pay me for the work, and that means the Jepsen library, test harness for each database, and all the reports are free for everyone. :)
  - eternalban 6y ago
    Loved the precise, concise, and complete technical expression in the Background section. (Thank you for sharing your great work.)
  - bullen 6y ago
    Does the Jepsen test require you to host the databases or can you test a system completely remotely?
    - aphyr 6y ago
      Sure, I could run Jepsen against anything I can talk to over the network, but a.) it might be painful or impossible to get a fresh, healthy cluster for each test run (It's... fairly common that databases stop being databases during a test) and b.) doing fault injection means I need a way to, well, inject faults. That's why Jepsen tests almost always involve Jepsen controlling the cluster.
      - bullen 6y ago
        Ok, what are the API calls you need now, and would need to remotely inject faults? Can you make a HTTP server API with a client implementation for it so that we can just make implement the server API and point your test cluster to the nodes for automatic testing? I understand this would be counter to you making your living out of this, but think of it as a "light" pro-bono version!
        deleted 6y ago
        [deleted]
        aphyr 6y ago
        It sounds like you're asking for contract work; you're welcome to email me at aphyr@jepsen.io.
nick238 6y ago
Does Dgraph work in geo-distributed applications? It's great to be "distributed" and all, but if your cluster is in one data center and it loses connection, you're hosed. I'd assume for something using Raft (and whatever other custom protocol you're also using) there must be a way to configure election timeouts, but I can't find any docs on it.
- mrjn 6y ago
  Depends upon how far apart are the servers. It is on our roadmap to experiment with Dgraph servers as part of one cluster across US based datacenters. Once we establish that, we can try it out across US / Europe, etc. We use 100ms heartbeats, with 20x that for election timeouts, in line with Etcd. Don't think that's exposed to the end user -- nobody has needed to tweak that.
mro_name 6y ago
Hi, was nosing around your website and find it very much appealing. Am currently for hire, rooted in germany. Whom should I contact? Thx! Am author of https://github.com/mro/librdf.sqlite https://github.com/mro/librdf.sqlite
- mrjn 6y ago
  We are looking for a Community Engineer: https://discuss.dgraph.io/t/hiring-community-engineer-to-start-immediately/6344 https://discuss.dgraph.io/t/hiring-community-engineer-to-sta...
mavbo 6y ago
I've been using Dgraph in building some PoCs and it's been great. My current confusion is around the push for a plain GraphQL compliant query endpoint as I'm not sure what the intended purpose is. Presumably it's not for direct connections between a front-end web client and the DB using something like Apollo Client. Should front ends still be going through some sort of API server that ingests GraphQL or REST style requests and queries Dgraph to build results? That's what I'm currently doing to implement things such as custom logic, side effects, authentication, etc. I'm not seeing the benefit of the plain GraphQL endpoint in that case, since I would be managing a separate GraphQL schema in the DB that is likely a subset or is similar to the API server's GraphQL schema. I've been sticking with GraphQL+- since it works great and I need an API server in between the front end and DB anyways. I've built some helper functions to map plain GraphQL request trees into expanded GraphQL+- queries for expanding edges since my API schema closely matches my Dgraph schema. Would love some insight into the plans or use case for the plain GraphQL endpoint.
- mrjn 6y ago
  The latest version of Dgraph 20.03.1 contains GraphQL endpoint. We're releasing few more features very soon, which would add object level auth, ability to trigger custom logic, call remote GraphQL endpoints to execute part of the query, and so on. So, you could put Dgraph directly in front of a frontend, without worry. Dgraph can also be called via Apollo Gateway, as part of their federation. We also have other very interesting stuff in the pipeline, which would cater a lot to front end development as well as backend. So, stay tuned.

by mrjn 6y ago

bullen 6y ago

Did you have to pay Kyle for him to test your database?

alexchamberlain 6y ago
The report states the work was funded by Dgraph and conducted in line with their ethics policy.
sorokod 6y ago
You may find the Ethics [1] section useful. Also, are you an acquaintance of Kyle Kingsbury? Just curious as you refer to him by his first name. [1] https://jepsen.io/ethics https://jepsen.io/ethics
- dang 6y ago
  It's an internet convention to refer to people by their first names.
aphyr 6y ago
Yep! Testing databases is my full time job. Vendors pay me for the work, and that means the Jepsen library, test harness for each database, and all the reports are free for everyone. :)
- eternalban 6y ago
  Loved the precise, concise, and complete technical expression in the Background section. (Thank you for sharing your great work.)
- bullen 6y ago
  Does the Jepsen test require you to host the databases or can you test a system completely remotely?
  - aphyr 6y ago
    Sure, I could run Jepsen against anything I can talk to over the network, but a.) it might be painful or impossible to get a fresh, healthy cluster for each test run (It's... fairly common that databases stop being databases during a test) and b.) doing fault injection means I need a way to, well, inject faults. That's why Jepsen tests almost always involve Jepsen controlling the cluster.
    - bullen 6y ago
      Ok, what are the API calls you need now, and would need to remotely inject faults? Can you make a HTTP server API with a client implementation for it so that we can just make implement the server API and point your test cluster to the nodes for automatic testing? I understand this would be counter to you making your living out of this, but think of it as a "light" pro-bono version!
      - deleted 6y ago
        [deleted]
      - aphyr 6y ago
        It sounds like you're asking for contract work; you're welcome to email me at aphyr@jepsen.io.

nick238 6y ago

Does Dgraph work in geo-distributed applications? It's great to be "distributed" and all, but if your cluster is in one data center and it loses connection, you're hosed. I'd assume for something using Raft (and whatever other custom protocol you're also using) there must be a way to configure election timeouts, but I can't find any docs on it.

mrjn 6y ago
Depends upon how far apart are the servers. It is on our roadmap to experiment with Dgraph servers as part of one cluster across US based datacenters. Once we establish that, we can try it out across US / Europe, etc. We use 100ms heartbeats, with 20x that for election timeouts, in line with Etcd. Don't think that's exposed to the end user -- nobody has needed to tweak that.

mro_name 6y ago

Hi, was nosing around your website and find it very much appealing. Am currently for hire, rooted in germany. Whom should I contact? Thx! Am author of https://github.com/mro/librdf.sqlite https://github.com/mro/librdf.sqlite

mrjn 6y ago
We are looking for a Community Engineer: https://discuss.dgraph.io/t/hiring-community-engineer-to-start-immediately/6344 https://discuss.dgraph.io/t/hiring-community-engineer-to-sta...

mavbo 6y ago

I've been using Dgraph in building some PoCs and it's been great. My current confusion is around the push for a plain GraphQL compliant query endpoint as I'm not sure what the intended purpose is. Presumably it's not for direct connections between a front-end web client and the DB using something like Apollo Client. Should front ends still be going through some sort of API server that ingests GraphQL or REST style requests and queries Dgraph to build results? That's what I'm currently doing to implement things such as custom logic, side effects, authentication, etc. I'm not seeing the benefit of the plain GraphQL endpoint in that case, since I would be managing a separate GraphQL schema in the DB that is likely a subset or is similar to the API server's GraphQL schema. I've been sticking with GraphQL+- since it works great and I need an API server in between the front end and DB anyways. I've built some helper functions to map plain GraphQL request trees into expanded GraphQL+- queries for expanding edges since my API schema closely matches my Dgraph schema. Would love some insight into the plans or use case for the plain GraphQL endpoint.

mrjn 6y ago
The latest version of Dgraph 20.03.1 contains GraphQL endpoint. We're releasing few more features very soon, which would add object level auth, ability to trigger custom logic, call remote GraphQL endpoints to execute part of the query, and so on. So, you could put Dgraph directly in front of a frontend, without worry. Dgraph can also be called via Apollo Gateway, as part of their federation. We also have other very interesting stuff in the pipeline, which would cater a lot to front end development as well as backend. So, stay tuned.