4 ms·

ArangoDB always makes for exciting benchmark posts. I could see myself there in a bowler hat with a fistful of racing chits screaming “go, Postgres, go.” I’d

by etxm 9y ago

ArangoDB always makes for exciting benchmark posts. I could see myself there in a bowler hat with a fistful of racing chits screaming “go, Postgres, go.” I’d love to see a competition were the developers of each database got to use the same hardware and data then tune the hell out of their configs, queries, and indices. Red Bull could sponsor it. I’d buy a T-shirt.

jsteemann 9y ago
Nice idea, I would take part in something like that too. By the way, there seems to be a raffle of free T-shirts (quite biased, as there's only the ArangoDB logo on them) in exchange for taking part in a survey: https://arangodb.com/community-survey/ https://arangodb.com/community-survey/
y4mi 9y ago
That sound ridiculously good. Like esports, for DBAs
- etxm 9y ago
  We could call it DB sports. On second thought that might be a bad name
kbenson 9y ago
That doesn't sound that hard to start. Something like RealWorld[1] and the Web Framework Benchmarks[2] combined but for DB workloads. Have one dataset that includes data amenable to OLAP and OLTP, but have separate tests each consisting of OLAP queries, OLTP queries, and combined queries. Choose a low-end, mid-range and high-end set of AWS or GCE instances/configs to normalize against. Let people submit pull requests with new technologies or configs. You'd want to get some funding to run the tests (or maybe solicit Google or Amazon to see if you could get the instance time donated once a month or something. If you started small, with maybe a portion of these features, and then scaled up over time, you might actually get to the point where you had tests that emulated a power failure, or master/slave and dual master scenarios and how they handle certain common network errors (split-brain). That would be an amazing resource. Edit: It occurs to me I probably should have read more of the article, since this is sort of what they are doing already... 1: https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld 2: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- odammit 9y ago
  Yeah after I posted it I started thinking about what it would take and what that would actually look like... and how you’d cheat :) It would probably require a few different categories with some sort of output assertion to validate the query performed right and a means of tracking CPU, usage ram usage, and execution time. It would be cool to see things like disaster recovery and chaos proofing as well.

by etxm 9y ago

jsteemann 9y ago

Nice idea, I would take part in something like that too. By the way, there seems to be a raffle of free T-shirts (quite biased, as there's only the ArangoDB logo on them) in exchange for taking part in a survey: https://arangodb.com/community-survey/ https://arangodb.com/community-survey/

y4mi 9y ago

That sound ridiculously good. Like esports, for DBAs

etxm 9y ago
We could call it DB sports. On second thought that might be a bad name

kbenson 9y ago

That doesn't sound that hard to start. Something like RealWorld[1] and the Web Framework Benchmarks[2] combined but for DB workloads. Have one dataset that includes data amenable to OLAP and OLTP, but have separate tests each consisting of OLAP queries, OLTP queries, and combined queries. Choose a low-end, mid-range and high-end set of AWS or GCE instances/configs to normalize against. Let people submit pull requests with new technologies or configs. You'd want to get some funding to run the tests (or maybe solicit Google or Amazon to see if you could get the instance time donated once a month or something. If you started small, with maybe a portion of these features, and then scaled up over time, you might actually get to the point where you had tests that emulated a power failure, or master/slave and dual master scenarios and how they handle certain common network errors (split-brain). That would be an amazing resource. Edit: It occurs to me I probably should have read more of the article, since this is sort of what they are doing already... 1: https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld 2: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/

odammit 9y ago
Yeah after I posted it I started thinking about what it would take and what that would actually look like... and how you’d cheat :) It would probably require a few different categories with some sort of output assertion to validate the query performed right and a means of tracking CPU, usage ram usage, and execution time. It would be cool to see things like disaster recovery and chaos proofing as well.