3 ms·
That doesn't sound that hard to start. Something like RealWorld[1] and the Web Framework Benchmarks[2] combined but for DB workloads. Have one dataset that in
by kbenson 9y ago
That doesn't sound that hard to start. Something like RealWorld[1] and the Web Framework Benchmarks[2] combined but for DB workloads. Have one dataset that includes data amenable to OLAP and OLTP, but have separate tests each consisting of OLAP queries, OLTP queries, and combined queries. Choose a low-end, mid-range and high-end set of AWS or GCE instances/configs to normalize against. Let people submit pull requests with new technologies or configs.
You'd want to get some funding to run the tests (or maybe solicit Google or Amazon to see if you could get the instance time donated once a month or something.
If you started small, with maybe a portion of these features, and then scaled up over time, you might actually get to the point where you had tests that emulated a power failure, or master/slave and dual master scenarios and how they handle certain common network errors (split-brain). That would be an amazing resource.
Edit: It occurs to me I probably should have read more of the article, since this is sort of what they are doing already...
1: https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld
2: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
- odammit 9y agoYeah after I posted it I started thinking about what it would take and what that would actually look like... and how you’d cheat :) It would probably require a few different categories with some sort of output assertion to validate the query performed right and a means of tracking CPU, usage ram usage, and execution time. It would be cool to see things like disaster recovery and chaos proofing as well.